A Behavioral Economic Reward Index Predicts Drinking Resolutions: Moderation Revisited and Compared With Other Outcomes

Acknowledgement: Mary J. Vignolo is now affiliated with the American College of Rheumatology Research and Education Foundation, Atlanta, GA. Andrew O. Westfall is now affiliated with the Department of Obstetrics and Gynecology, University of Alabama at Birmingham School of Medicine, and the Centre for Infectious Disease Research, Lusaka, Zambia.

This research was supported in part by Grants R01 AA008972 and K02 AA000209 from the National Institute on Alcohol Abuse and Alcoholism. The authors thank Paula D. Rippens, Bethany C. Black, and H. Russell Foushee for their contributions to the data collection phase of the research, and G. Alan Marlatt for his comments on an earlier version of this article. Portions of this research were presented at the Third Annual Workshop Mechanisms of Behavior Change in Behavioral Treatment held at the annual meeting of the Research Society on Alcoholism, Chicago, IL, July 2007.

Behavioral economic models of choice behavior have been widely applied to an analysis of substance misuse and other addictive behaviors in humans (e.g., Bickel & Marsch, 2001; Green & Kagel, 1996; Vuchinich & Tucker, 1996, 1998). Behavioral economics involves a merger of operant approaches to understanding choice behavior, particularly impulsive choice (Ainslie, 1975), with microeconomic models of consumer behavior (Rachlin, Battalio, Kagel, & Green, 1981). Both focus on how individuals allocate limited resources such as time, money, and behavior to obtain commodities available at different costs and over different delays, and strength of preference for a given commodity (e.g., drug use) is inferred from the relative resources or behavior allocated to obtain it (Premack, 1965; Rachlin, 1971). For example, the well-established matching law quantifies how humans and animals alike distribute or “match” relative response rates in proportion to the relative rates of reinforcement available from different activities (Herrnstein, 1970).

This approach is well suited to studying demand for drugs in relation to other commodities available in the natural environment (Vuchinich & Tucker, 1996, 1998). Behavioral economic models view substance misuse as a persistent preference for short-term rewards and a devaluation of larger, delayed rewards that support adaptive functioning. Research has consistently shown that preference for substance use decreases as constraints on access to the substance increase and as constraints on access to valued non-drug-related alternatives decrease (Vuchinich & Tucker, 1998). Moreover, persons with addictive behavior problems tend to devalue, or discount, delayed rewards more than normal controls (Bickel & Marsch, 2001). Control of their current behavior is less sensitive to delayed consequences, such as the adverse long-term effects of substance use.

As applied to attempts to resolve alcohol problems, these findings suggest that problem drinkers with greater sensitivity to longer term contingencies, even when drinking heavily, should have a better prognosis and that shifting control of behavior from shorter to longer term contingencies should promote resolution stability. Our earlier prospective studies of resolution attempts by community-dwelling treated and untreated problem drinkers supported this hypothesis (Tucker, Foushee, & Black, 2008; Tucker, Vuchinich, Black, & Rippens, 2006; Tucker, Vuchinich, & Rippens, 2002). Shortly after initiation of abstinence or problem-free moderation drinking, participants reported their monetary expenditures on alcoholic beverages and other commodities during the year before resolution onset using an expanded Timeline Followback (TLFB) interview (Sobell & Sobell, 1992). Establishing relative preference for alcohol through allocation of monetary resources is based on experimental work showing that the relative values of different, concurrently available commodities can be quantified by measuring choice among the commodities under varying constraints (Herrnstein, 1970; Premack, 1965). Many different activities are available in the natural environment, and monetary allocation offers a common metric to assess their relative reinforcement value (Vuchinich, Tucker, & Harllee, 1988).

Our main research focus was on studying successful natural recoveries achieved without treatment. Natural recovery samples typically include middle-income to upper income individuals (Sobell, Sobell, & Toneatto, 1992) who have complex, fixed, and recurring expenditures (e.g., mortgages, automatic payroll deductions) as well as considerable discretionary expenditures (e.g., for recreation, alcohol, voluntary savings). Because their preferences should be more readily expressed within discretionary, as opposed to fixed, spending patterns, the proportion of discretionary spending on alcoholic beverages was compared with money put into savings for future use, which was conceptualized as representing the value of rewards available over shorter and longer time horizons, respectively. Greater relative allocation to savings than to drinking, reflected in lower values on this Alcohol-Savings Discretionary Expenditure (ASDE) index, was viewed as indicating higher relative preferences for delayed rewards made possible by savings compared with more immediate rewards from drinking.

As hypothesized, problem drinkers who maintained stable resolutions had lower preresolution ASDE values than those who had unstable resolutions and relapsed at any point during the 1- to 2-year follow-ups (Tucker et al., 2002, 2006, 2008). The ASDE index had unique incremental utility in predicting stable versus unstable resolutions after controlling for established outcome predictors (e.g., problem severity, drinking practices), and it had predictive utility across intervention-naive and intervention-exposed resolution groups (Tucker et al., 2006). In addition to predicting long-term resolution stability, the ASDE index predicted drinking patterns during the early months of the postresolution period in a study that implemented Interactive Voice Response (IVR) self-monitoring with recently resolved, untreated problem drinkers (Tucker et al., 2008).

This research showed that contextually sensitive measures of the reward value of drinking in relation to other activities added unique information in an account of resolution outcomes. However, in these studies many more participants achieved stable abstinent than nonabstinent resolutions, so predictors of moderation apart from abstinence could not be investigated in the studies individually. A more extensive analysis with a larger sample that includes more participants who drank in a sustained nonproblem manner is needed to examine specific predictors of moderation. This issue has gained renewed importance as interventions continue to expand beyond abstinence-oriented treatments for alcohol-dependent persons, to include population-based public health interventions for the untreated majority with less severe problems for whom moderation is a more common and acceptable outcome (Tucker, 2003). Moderation outcomes are more common among untreated problem drinkers who quit on their own compared with the minority who seek treatment, partly because treatment seekers have more serious problems. Although early treatment research found moderation to be associated with lower problem severity, younger age, and stable life circumstances (reviewed by Miller & Munoz, 2005; Rosenberg, 2004), there have been few recent advances, with the exception that higher self-efficacy to resist drinking in high risk situations has been associated with moderation outcomes (Saladin & Santa Ana, 2004).

To obtain a sufficient sample to investigate stable moderation apart from other outcomes, we pooled the data from our three prior studies and conducted new analyses to evaluate the utility of the ASDE index in distinguishing stable nonabstinent resolutions from stable abstinent resolutions and unstable resolutions that involved problem drinking at some point over the 1- to 2-year follow-ups. Our interest in examining this issue in a re-analysis comes from early theorizing about the processes involved in moderation (Marlatt, 1985) and from preliminary findings in our IVR study (Tucker et al., 2008) that supported the theorizing. Over two decades ago, Marlatt (1985, pp. 329–344) raised the interesting but still unstudied hypothesis that abstinence and relapse are opposite ends of the same dynamic behavioral regulation process, reflecting over- and undercontrol of the daily act of drinking, respectively. Moderation was thought to involve a different regulation process that depends on “lifestyle balance” and repetitive choices to drink well within the boundaries of extreme restraint or loss-of-control drinking. To the extent that the ASDE index is a functional measure of preference for alcohol in relation to delayed rewards made possible by savings, one would expect successful moderate drinkers to organize their behavioral allocation (tracked via financial expenditures) over longer intervals compared with those who relapse or abstain. Framing Marlatt's (1985) “differential regulation” hypothesis within behavioral economic theory, lower ASDE values, reflecting more balanced monetary allocations between short-term and longer term objectives (i.e., drinking and saving for the future), should predict moderation compared with other outcomes.

The pooled data set included 30 drinkers with moderation outcomes, which was sufficient to evaluate the hypothesis that stable resolutions involving some moderation drinking over 1 to 2 years would be predicted by lower preresolution ASDE values compared with other outcomes. After this primary behavioral economic hypothesis was evaluated, established moderation predictors assessed at baseline, including problem severity, alcohol dependence, and self-efficacy, were included with the ASDE index in multinomial logistic regression models to determine if the index had unique incremental predictive utility in distinguishing outcomes among participants who resumed drinking (relapse or moderation) and among those who remained resolved (abstinence or moderation). A series of multivariable models were used to subject the ASDE index to a rigorous systematic evaluation after controlling for multiple covariates and to maintain favorable events-to-predictor ratios in each model (Peduzzi, Concato, Kemper, Holford, & Feinstein, 1996) within the limits set by the number of moderation cases in the pooled data set. We hypothesized that the ASDE index would provide significant incremental predictive utility over predictors suggested by prior research and would be particularly sensitive for distinguishing moderated and relapsed outcomes—that is, the differential regulation processes theorized by Marlatt (1985) and assessed by our behavioral economic index should be most apparent among participants who engaged in some postresolution drinking.

Sample Selection and Characteristics

Participants were recruited from the community via media advertisements in metropolitan areas in Alabama, Florida, Georgia, Mississippi, and Tennessee. The advertisements asked for research volunteers who had recently overcome a drinking problem with or without treatment. Respondents to the advertisements called a toll-free number, received a description of the research, and were screened with the Michigan Alcoholism Screening Test (MAST; Selzer, 1971), Alcohol Dependence Scale (ADS; Skinner & Horn, 1984), and Drinking Problems Scale (DPS; Cahalan, 1970). Eligible participants were scheduled for interviews in a place convenient for them. All studies were conducted in compliance with university institutional review board and American Psychological Association ethical standards for research with humans. Participants were informed that the research was covered by a confidentiality shield issued by the U.S. Department of Health and Human Services.

Eligibility criteria included a minimum 5-year drinking problem history (M = 16.70 years, SD = 9.31), no current other drug misuse (except nicotine), and recent cessation of problem drinking (M = 3.93 months resolved, SD = 1.78). Resolution onset was defined as the most recent date that participants began abstaining or drinking in a nonproblem manner without further heavy drinking. At all assessments, moderation was determined using criteria associated with low health risks related to drinking (Sobell et al., 1992): (a) <55 g (70 ml) of 190-proof ethanol consumed per drinking day; (b) no dependence symptoms (as assessed by the ADS); and (c) no alcohol-related negative health, psychosocial, vocational, financial, or legal consequences (as assessed by the DPS). These criteria are consistent with other drinking guidelines (e.g., National Institute on Alcohol Abuse and Alcoholism [NIAAA], 2005; World Health Organization, 2000) that generally set upper limits at ≤4 drinks/day for men and ≤3 drinks/day for women.

All of the studies included untreated problem drinkers who had initiated natural resolutions, and Tucker et al. (2006) also included a group that had received alcohol treatment from a qualified provider or attended more than two Alcoholics Anonymous meetings within about 12 months of resolution onset. Two studies (Tucker et al., 2002, 2006) required an initial resolution of 2 to 6 months and had a 2-year follow-up; the third study that involved IVR self-monitoring (Tucker et al., 2008) required a shorter initial resolution of 1 to 3 months and had a 1-year follow-up. Otherwise, the studies used identical selection criteria and follow-up procedures. Summed across studies, 205 (81.03%) of the 253 initially enrolled participants completed the 1-year follow-up required for inclusion in the pooled sample; 202 provided useable income and expenditure data and were included in the data analyses. Attrition was due to participant withdrawal or lost contact (42), significant discrepancies between participant and collateral reports of drinking (5), or death (1).

Although not an inclusion criterion, all participants met third- (Tucker et al., 2002) or fourth- (Tucker et al., 2006, 2008) edition Diagnostic and Statistical Manual of Mental Disorders criteria for alcohol dependence (American Psychiatric Association, 1987, 1994). As shown in Table 1, ADS scores fell in the moderate to low substantial dependence range (Skinner & Horn, 1984). Consistent with research showing that seeking help is associated with more severe problems, ADS, MAST, and DPS scores were relatively higher in the one study that included participants with a help-seeking history (Tucker et al., 2006). Otherwise, no differences were found across studies in preresolution drinking practices, postresolution outcomes, demographic characteristics, and preresolution income and expenditures on alcoholic beverages and money put into savings.

Sample Characteristics at Initial Assessment as a Function of Resolution Status at Follow-Up

Because our goal was to predict stable moderation apart from other outcomes, we classified participants conservatively into mutually exclusive groups based on drinking practices and problems over the entire follow-up. Those who abstained or drank moderately without problems at all follow-up points were considered to have stable resolutions, either resolved abstinent (RA) or resolved nonabstinent (RNA). Those who engaged in any problem drinking were considered to have unstable resolutions (UR), even if they later abstained or moderated.

Table 1 summarizes the sample characteristics as a function of participants' postresolution drinking status based on all follow-up data. During the initial resolution period required for inclusion, 89.1% of participants had abstained continuously and 10.9% had engaged in moderate drinking. Most participants' current drinking goal choice was informed by personal experience; 85% had made one or more serious resolution attempts in the past, with moderation attempts outnumbering abstinence attempts by about 4:1. During the present 1- to 2-year follow-up, 47.0% of participants maintained continuous or nearly continuous abstinence, 14.9% engaged in some moderate drinking with no problem drinking, and 38.1% engaged in problem drinking at some point. Of those who drank moderately, women consumed a mean of 27.5 ml of 190-proof ethanol per drinking day (SD = 8.56), and men consumed a mean of 38.9 ml (SD = 19.0), which fall within NIAAA (2005) gender-adjusted guidelines for low-risk drinking. Of those who drank heavily, over half relapsed during the first postresolution year. Mean quantities consumed per postresolution drinking day were 93.8 ml (SD = 51.1) for women and 123.8 ml (SD = 75.0) for men. During the first postresolution year, the mean and median number of drinking days for RNA participants was 73.93 (SD = 111.51) and 5.50 days, respectively, and the mean and median for UR participants was 44.23 (SD = 67.23) and 20.0 days, respectively.

Procedure

A trained interviewer conducted 1.5- to 3.0-hour individual interviews at baseline and at the annual follow-up points. After giving written informed consent, participants were administered a noninvasive breath test (Alco-Sensor III; Intoximeters, Inc., St Louis, MO) to verify sobriety. All predictors were derived from the initial interview, which covered drinking practices, life contexts, and monetary allocation during the year before participants' recent resolution up to the time of the interview. The follow-up assessments covered the time since the last interview. Brief phone interviews conducted midway between the annual follow-ups assessed drinking and help-seeking status and maintained contact. Participants received $40 for each annual interview and completion of questionnaires that they returned by mail, $10 for each phone interview, and a $50 bonus if they completed all assessments. The procedures that provided the data for the analyses are summarized in the next section and in Tucker et al. (2002, 2006, 2008).

Drinking practices and money spent on alcohol

Established TLFB procedures (Sobell & Sobell, 1992) were used to assess daily drinking practices during the preresolution year and again at each annual follow-up point. Participant reports of ounces of beer, wine, and liquor intake were converted to milliliters of 190-proof ethanol for analysis. Participants also reported how much money they spent each day on alcoholic beverages, regardless of whether the beverages were consumed. This was not excessively difficult because alcoholic beverages are sold in standard quantities, and problem drinkers typically buy and consume large quantities of a limited range of preferred beverages. As needed, TLFB interviewing techniques were used to facilitate reports of money spent on alcohol (e.g., use of anchor events, identification of sustained behavior patterns).

Monetary allocation

Participants reported their monetary income and expenditures during the same periods using an expanded set of commodity classes derived from U.S. federal consumer expenditure surveys (Vuchinich & Tucker, 1996). They were instructed to bring in financial records (e.g., bank records, paycheck stubs), and documented information was recorded first; 59.4% of participants provided some financial records. Then TLFB interviewing techniques were used to complete the financial assessment. Income in dollars was reported by source (e.g., work income, unemployment benefits, pensions, loans). Expenditures were reported in three general categories, each with subcategories, including housing (e.g., mortgage, rent, utilities), consumable goods (e.g., food, tobacco, alcohol), and other (e.g., entertainment, transportation, loan payments, money saved). Reports in each category typically involved many transactions during the assessment interval, which were summed to obtain category totals for analysis. In addition to direct verification using financial records, internal consistency and reliability checks on participants' reports of monetary allocation patterns supported their accuracy (reported in Tucker et al., 2006).

As described in Tucker et al. (2002, 2006, 2008), expenditures during the preresolution year were separated into obligatory and discretionary categories. Obligatory expenditures were for essential, ongoing, and largely fixed costs of living, including housing, food, transportation, medical, loan, and automatic payroll deductions (e.g., taxes, retirement, health insurance). Discretionary expenditures were for less essential commodities that could be purchased intermittently, including recreation, entertainment, alcohol, tobacco, other consumable goods, gifts, and elective savings. The ASDE index was computed as the proportion of discretionary expenditures summed over the preresolution year spent on alcohol minus the proportion of preresolution discretionary expenditures put into savings. ASDE values could range from 1.0 to –1.0, with lower scores representing proportionally less spending on alcohol and more on savings.

Questionnaires

After each interview, participants completed questionnaires that assessed moderation predictors in addition to those assessed during screening. Self-efficacy expectations to resist urges to drink heavily in high-risk situations were assessed using the Situational Confidence Questionnaire (SCQ; Annis & Graham, 1988), and health status was assessed using the health portion of the Health and Daily Living Form (HDL; Moos, 1985). Questionnaires were scored using established methods. Table 1 presents the total scores from the initial assessment.

Checks on data quality

In every study, in addition to checks on participants' financial reports, their reports relevant to the inclusion criteria and follow-up drinking status were assessed through collateral interviews or participant reliability checks when collaterals were unavailable. These data, summarized here, were reported previously (Tucker et al., 2002, 2006; Tucker, Foushee, Black, & Roth, 2007). Summed across studies, collaterals were interviewed at least once for 75.61% of the enrolled sample. Participant data were excluded when collaterals failed to verify participant reports relevant to the inclusion criteria or their drinking status during the follow-up. This rarely occurred (less than 2% of the initial sample of 253). For participants' retained for analysis, good to excellent agreement levels were found for drinking dimensions that could be directly observed by collaterals (e.g., alcohol-related problems, types of beverages consumed, date of initial resolution). The reliability of participant reports of drinking practices and money spent on alcohol also was examined for participants in the IVR study and was found to be excellent (Tucker et al., 2007). These findings strongly suggest that participants retained in the sample were reporting accurately.

Statistical Analysis

The mutually exclusive outcome groups were based on participants' drinking practices and problems over the entire follow-up interval: RA (n = 95)—continuous abstinence; RNA (n = 30)—some low-risk drinking with no problem drinking; or UR (n = 77)—one or more drinking episodes that exceeded the moderation criteria at any point. These conservative operational definitions deliberately separated recovering problem drinkers who resumed alcohol consumption into outcome groups on the basis of whether they engaged in any high-risk drinking, regardless of their terminal outcome status.

To evaluate the main behavioral economic hypotheses, the preresolution year monetary allocations to alcohol and savings, computed as a proportion of discretionary expenditures, were first examined as a function of drinking outcome status in a 3 (outcome status) × 2 (allocation type) analysis of variance (ANOVA) with repeated measures on the second factor. The Outcome × Allocation interaction effect from this analysis was used to determine whether the difference in these allocations, which constituted the ASDE index, was significantly related to outcome group membership. Once confirmed, we used a series of three-group multinomial logistic regression analyses to examine the utility of the ASDE index in predicting outcome group membership in relation to established predictors, including measures of problem severity (MAST, ADS, HDL physical health subscale, problem duration, help-seeking history), TLFB reports of preresolution drinking (days well functioning [abstinent and light drinking days combined], mean milliliters of ethanol consumed per drinking day), self-efficacy to resist heavy drinking (SCQ), and demographic characteristics.

Because these analyses focused on identifying predictors of moderated outcomes, the RNA group was the referent group so that the results yielded RA versus RNA and UR versus RNA contrasts and associated odds ratios (ORs) that indicated effect sizes. Although the limited RNA participants (30) prohibited comprehensive multivariable models that included all predictors simultaneously (Peduzzi et al., 1996), the RNA sample was sufficient to maintain a favorable event-to-variable ratio in a series of multinomial logistic regressions that included, first, the ASDE index alone, and then, in subsequent models, the ASDE index plus one other predictor. The latter analyses determined if the predictive utility of the ASDE index was independent of the predictive effects of each established predictor. Significant effects from the two-predictor models were then used to construct three-predictor models that evaluated whether the ASDE index continued to predict RNA outcomes beyond significant problem severity and drinking quantity measures. A final four-variable model examined whether the ASDE continued to have unique predictive utility when three other significant predictors from the three-variable models were included simultaneously. Continuous predictor variables in all logistic regression models were standardized to have a mean of 0 and a standard deviation of 1. The ORs and associated 95% confidence intervals (CIs) were based on a 1-standard deviation change in the predictors and allowed direct comparisons across predictors.

Tests of the Behavioral Economic Hypotheses

Table 1 summarizes univariate differences between the three outcome groups for established moderation predictors, and Table 2 summarizes group differences for the expenditure data from the preresolution year, including the ASDE index and expenditure components from which it was derived. The 3 × 2 ANOVA on the proportions of discretionary expenditures indicated an expected allocation main effect, F(1, 199) = 78.56, p < .0001, which reflected greater overall proportional allocation to drinking than savings prior to resolution. More important, as shown in Figure 1, a significant interaction effect was obtained that supported the hypotheses, F(2, 199) = 11.12, p = .0001. Comparisons using Tukey's honestly significant difference test showed that, as predicted, RNA participants had significantly lower discrepancies between alcohol and savings allocation proportions than both UR participants (p < .05) and RA participants (p < .05). RNA participants allocated proportionally less discretionary spending to alcohol and more to savings compared with UR participants (ps < .05) and less to alcohol than RA participants (p < .05).

Behavioral Economic Variables Based on Preresolution Monetary Allocation Patterns as a Function of Resolution Status at Follow-Up

Figure 1. Resolution Status × Allocation Type interaction among components of the Alcohol-Savings Discretionary Expenditure (ASDE) index based on the proportion of discretionary expenditures for alcohol and savings during the year prior to resolution onset (y axis). The error bars represent the standard errors of the drinking outcome group means.

The multinomial logistic regression analysis that included the ASDE index as the sole predictor revealed significant effects for both the UR versus RNA contrast (OR = 2.89, 95% CI = 1.77, 4.73, p < .0001) and the RA versus RNA contrast (OR = 1.93, 95% CI = 1.23, 3.02, p = .004). The OR from the first contrast indicated that a 1-standard-deviation increase in the ASDE index was associated with a 2.89-fold increase in the odds of resuming problem drinking compared with stable moderation. The OR from the second contrast indicated that a 1-standard-deviation increase in the ASDE index was associated with a 1.93-fold increase in the odds of stable abstinence compared with moderation.

The preceding logistic regression used drinking outcome assignments based on all available data from the 202 participants who had an ASDE score and at least 1 year of follow-up data. When this analysis was restricted to participants who were followed for 2 years (n = 152) and thus provided the longest continuous behavioral records, the ASDE index remained a significant predictor of both the UR versus RNA contrast (OR = 2.79, CI = 1.56, 5.00, p = .0006) and the RA versus RNA contrast (OR = 1.95, CI = 1.14, 3.32, p = .014). The same pattern of results was observed in an additional sensitivity analysis (n = 193) that excluded 4 RNA and 5 UR participants who were mostly abstinent but occasionally drank either moderately or heavily (UR vs. RNA: OR = 2.55, CI = 1.54, 4.21, p = .0003; RA vs. RNA: OR = 1.74, CI = 1.10, 2.75, p = .017). Consistent with the main behavioral economic hypothesis, frequent moderate drinkers had the lowest and frequent heavy drinkers had the highest mean ASDE scores.

Overall, these results supported the hypotheses concerning the ASDE index. The composite index separated the RNA group from the RA and UR groups, which were more similar.

Predictive Utility of the ASDE Index Relative to Established Moderation Predictors

Table 3 summarizes the results of logistic regressions that included the ASDE index and one other moderation predictor and also presents the correlations between the ASDE index and the other predictors. Four findings are noteworthy. First, the ASDE index showed low to modest correlations with all other predictors, ranging from .00 to .36 (rs > .14 or < −.14 were significant at p < .05), indicating that the ASDE was largely unrelated to the other predictors and capable of contributing new information to the prediction of outcomes in the multivariable models. Second, the analyses replicated many established moderation predictors, including lower dependence, fewer psychosocial and health problems, shorter problem durations, lower quantities consumed on drinking days, absence of help seeking, stable sociodemographic characteristics, and higher self-efficacy. Third, when the two-predictor models were run, the ASDE index remained a robust predictor of outcomes among the subset of participants who drank during the follow-up. The UR versus RNA contrast was significant in all models that included another predictor, indicating that the ASDE index explained unique variance after accounting for the other predictor. Fourth, the ASDE index distinguished outcomes among the subset of participants who maintained resolution. The RA versus RNA contrast was significant for the index when it was included with another predictor in all but two models that included either the MAST or mean milliliters per drinking day.

Multinomial Logistic Regressions Using the ASDE Index and One Established Moderation Predictor

On the basis of these results, additional three-variable logistic regressions were conducted that included the ASDE index and two other variables of conceptual interest or empirical utility. The MAST and ADS were not included in the same model because they were highly correlated (r = .68, p < .001) and would, therefore, be largely redundant in the same model. Table 4 presents the results of the three-variable models that we examined. In all eight complex models, the ASDE index continued to separate the UR and RNA groups. The index was not highly effective in separating the RA and RNA groups, although trends that approached significance for the ASDE were observed in Models 1, 3, and 8. A positive help-seeking experience and mean milliliters ethanol/drinking day consistently separated all three outcome groups, with absence of help seeking and lower quantities consumed being associated with an RNA status. The MAST, ADS, and SCQ contributed significantly to the separation of the UR and RNA groups. The MAST and ADS also separated the RA and RNA groups in models that did not include drinks per drinking day, but inclusion of that variable attenuated their predictive utility for the RA–RNA contrast.

Multinomial Logistic Regression Models Using the ASDE Index With Two Moderation Predictors

A final, comprehensive four-variable model was constructed from the strongest predictors identified in Table 3, namely the MAST, SCQ, mean milliliters ethanol/drinking day, and ASDE index. For the UR–RNA contrast, the SCQ (OR = 0.30, CI = 0.12, 0.74, p = .009), mean milliliters ethanol/drinking day (OR = 7.26, CI = 1.77, 29.74, p = .006), and ASDE (OR = 2.46, CI = 1.27, 4.77, p = .008) were statistically significant unique predictors, whereas for the RA–RNA contrast, the MAST (OR = 1.96, CI = 1.03, 3.72, p = .04) and mean milliliters ethanol/drinking day (OR = 9.54, CI = 2.37, 38.33, p = .002) were the statistically significant unique predictors.

The findings add to evidence supporting a behavioral economic analysis of drinking resolutions and extend the utility of a measure of preference for alcohol derived from preresolution spending patterns to predict moderation. Stable resolutions involving moderate alcohol use over 1- to 2-year follow-ups were associated with proportionally more preresolution discretionary monetary allocation to savings and less to alcohol compared with continuously abstinent resolutions and unstable resolutions that involved some problem drinking. Lower ASDE values presumably reflect more balanced monetary allocations between short-term and longer term objectives, suggesting that the temporal intervals over which problem drinkers organize and allocate their behavior, even while drinking heavily, may help identify those most able to make a transition to stable moderate use.

The support for the ASDE index in this re-analysis of pooled data from prior studies was obtained in conjunction with results that replicated established moderation predictors. As found during the controlled drinking debate (cf. Marlatt, 1983), moderation was associated with greater social stability and with lower problem severity, including shorter drinking histories, lower alcohol dependence and quantities consumed per drinking day, and fewer alcohol-related psychosocial problems (Miller & Munoz, 2005; Rosenberg, 2004). Higher self-efficacy to resist heavy drinking in high-risk situations also predicted moderation, which replicated recent findings that added this variable to those identified during the controlled drinking debate (Saladin & Santa Ana, 2004).

When included in models with other significant predictors, the ASDE index added unique information to the prediction of moderation and was especially effective in distinguishing outcomes among participants who engaged in postresolution drinking. The ASDE index was significant for the RNA–UR contrast in all models evaluated. Drinkers who maintained moderation had lower ASDE, MAST, and ADS scores and higher self-efficacy scores than those who relapsed.

The ASDE index also predicted outcomes among participants who remained resolved, separating the abstinent and moderation groups in models that included the ASDE alone or with one other predictor. The index was less effective in separating these groups in complex models that included quantities consumed, an attenuation that may be due in part to heterogeneity in the RA group. Some abstainers may be able to drink moderately but have not exposed themselves to postresolution alcohol use, whereas others might resume problem drinking. This unobserved moderation versus relapsed outcome among abstainers can, therefore, limit the full predictive significance of individual predictors of moderation outcomes. Given this attenuated separation, for purposes of choosing an initial abstinence or moderation drinking goal, it seems prudent clinically to require multiple favorable indicators of likely success at moderation until further research can establish decision-making algorithms that satisfactorily separate all three outcome groups. The present findings suggest that supplementing the MAST, ADS, and SCQ with questions about consumption quantities on drinking days and money spent on alcohol and put into savings provides a sound basis for making clinical judgments about initial drinking goal choice.

The ASDE findings also have implications for behavioral economic research and addictive behavior change applications. Evidence is accumulating that addictive behaviors are characterized by a foreshortened view of the future and that successful behavior change will likely involve a shift from a shorter to a longer view of the future and organizing behavior accordingly. In addition to the present support based on money allocation patterns in the natural environment, behavioral economic research on temporal discounting of hypothetical money and health outcomes has consistently found steeper discount functions in smokers, problem drinkers, opiate addicts, and gamblers (Bickel & Marsch, 2001). Social psychological studies have similarly found that the time perspectives of substance abusers are more present oriented and less future oriented than for normal controls (Henson, Carey, Carey, & Maisto, 2006; Keough, Zimbardo, & Boyd, 1999). Addiction treatment outcomes also are predicted by behavioral impulsivity measures that span laboratory and naturalistic assessments, including delay discounting of hypothetical rewards (e.g., Yoon et al., 2007), demand curve analysis based on hypothetical alcohol purchase tasks (MacKillop & Murphy, 2007), questionnaire measures of relative reinforcement value (e.g., Murphy, Correia, Colby, & Vuchinich, 2005; Schmitz, Sayre, Hokanson, & Spiga, 2003), and experiential discounting tasks that assess delay discounting using real, rather than hypothetical, monetary rewards (Krishnan-Sarin et al., 2007).

Such measures of behavioral impulsivity developed within a behavioral economic framework guided by the matching law (Herrnstein, 1970) show inconsistent relationships with personality questionnaires that assess traitlike impulsive tendencies, and the latter measures have no utility in predicting treatment outcomes (e.g., Krishnan-Sarin et al., 2007). Although it remains to be determined the extent to which the behavioral economic measures assess common or different dimensions of behavioral impulsivity and the relative reinforcing efficacy of substances (MacKillop & Murphy, 2007; Reynolds, Richards, Horn, & Karraker, 2004), they appear to measure functional changes in preferences for substance use and nondrug alternatives that are at the core of the dynamic addictive process. The ASDE index as currently assessed is decidedly among the more molar behavioral economic measures and appears to measure the relative reinforcement value of alcohol in the context of resource allocation to commodities available over different temporal intervals. Although the ASDE may prove to be less sensitive to short-term preference shifts than are brief measures of the current demand for drugs, it provides a comprehensive benchmark based on behavior patterns in the natural environment against which to evaluate the utility of briefer measures of relative preferences, including laboratory preparations.

Regardless of whether foreshortened views of the future are a cause or consequence of the addictive process, or both, it seems likely that interventions may facilitate positive change by promoting contact with the set of delayed positive consequences, or the sober “consumption bundle,” that typically flows from a sober lifestyle. Behavior patterns with delayed positive consequences often are more valuable as a whole compared with discrete acts chosen day-to-day (Rachlin, 1995), and the likelihood of maintaining longer term, higher yield patterns is presumably greater once contact is made with the delayed positive consequences. Drawing attention to delayed consequences or signaling their future availability (e.g., via self-monitoring, motivational interviewing, or decisional balance exercises) is one way to shift behavioral organization toward the future. Such approaches may reduce the appeal of short-term discrete rewards, like substance use, by helping people frame choices as involving an extended series of linked behaviors, events, and outcomes with higher overall value (Chapman, 1996; Rachlin, 1995).

The present research has limitations that merit attention in future studies. First, despite pooling across studies, the number of participants with stable moderation resolutions was still relatively small, which limited the number of predictors that could be evaluated simultaneously in multivariate models. Although quite coherent across models, the results merit replication with a larger sample of moderate drinkers. Second, because two of the earlier studies (Tucker et al., 2002, 2006) included too few moderate drinkers to analyze them separately from abstainers, the positive ASDE findings in those studies may have been amplified by including RNA participants in the stable resolution group along with RA participants. However, RNA participants constituted less than 12% of the stable resolution groups in these studies, suggesting that the overall ASDE results and interpretation were appropriate. Third, future studies of resolution should expand the window of selection to include problem drinkers who have resolved quite recently or are contemplating doing so. Participants with drinking problems that fall short of clinical diagnostic criteria also should be studied, and drinking-related inclusion criteria should be relaxed to allow research on outcomes that fall short of stable moderation but entail substantial reductions in drinking and related harm.

Pursuing the public health implications of understanding pathways to and predictors of moderation will require expanding the scope of research to include new concepts and methods, such as those provided by behavioral economics. Better characterizing the moderate use of alcohol by individuals with a history of problem drinking may provide new insights about the behavior regulation processes involved in resolution and relapse and help guide innovative behavior change strategies that increase contact with the population with problems.

Ainslie, G. (1975). Specious reward: A behavioral theory of impulsiveness and self-control. Psychological Bulletin, 82, 463–496.

American Psychiatric Association. (1987). Diagnostic and statistical manual of mental disorders (3rd ed.). Washington, DC: Author.

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Annis, H. M., & Graham, J. M. (1988). Situational Confidence Questionnaire (SCQ-39) user's guide. Toronto, Ontario, Canada: Addiction Research Foundation.

Bickel, W. K., & Marsch, L. A. (2001). Toward a behavioral economic understanding of drug dependence: Delay discounting processes. Addiction, 96, 73–86.

Cahalan, D. (1970). Problem drinkers: A national survey. San Francisco: Jossey-Bass.

Chapman, G. B. (1996). Temporal discounting and utility for health and money. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 771–791.

Green, L., & Kagel, J. H. (Eds.). (1996). Advances in behavioral economics: Vol. 3. Substance use and abuse. Norwood, NJ: Ablex.

Henson, J. M., Carey, M. P., Carey, K. B., & Maisto, S. A. (2006). Associations among health behaviors and time perspective in young adults: Model testing with boot-strapping replication. Journal of Behavioral Medicine, 29, 127–137.

Herrnstein, R. J. (1970). On the law of effect. Journal of the Experimental Analysis of Behavior, 13, 243–266.

Keough, K. A., Zimbardo, P. G., & Boyd, J. N. (1999). Who's smoking, drinking, and using drugs? Time perspective as a predictor of substance use. Basic and Applied Social Psychology, 21, 149–164.

Krishnan-Sarin, S., Reynolds, B., Duhig, A. M., Smith, A., Liss, T., McFetridge, A., Cavallo, D. A., et al. (2007). Behavioral impulsivity predicts treatment outcome in a smoking cessation program for adolescent smokers. Drug and Alcohol Dependence, 88, 79–82.

MacKillop, J., & Murphy, J. G. (2007). A behavioral economic measure of demand for alcohol predicts brief intervention outcomes. Drug and Alcohol Dependence, 89, 227–233.

Marlatt, G. A. (1983). The controlled drinking controversy: A commentary. American Psychologist, 38, 1097–1110.

Marlatt, G. A. (1985). Lifestyle modification. In G. A.Marlatt & J. R.Gordon (Eds.), Relapse prevention: Maintenance strategies in the treatment of addictive behaviors (pp. 280–348). New York: Guilford.

Miller, W. R., & Munoz, R. F. (2005). Controlling your drinking: Tools to make moderation work for you. New York: Guilford.

Moos, R. H. (1985). Health and Daily Living Form. In D. J.Lettieri, J. E.Nelson, & M. A.Sayers (Eds.), Alcoholism treatment assessment research instruments (NIAAA Treatment Handbook Series 2, pp. 275–292). Washington, DC: U.S. Government Printing Office.

Murphy, J. G., Correia, C. J., Colby, S. M., & Vuchinich, R. E. (2005). Using behavioral theories of choice to predict drinking outcomes following a brief intervention. Experimental and Clinical Psychopharmacology, 13, 93–101.

National Institute on Alcohol Abuse and Alcoholism. (2005). Helping patients who drink too much: A clinician's guide. Rockville, MD: U.S. Department of Health and Human Services, National Institutes of Health, National Institute on Alcohol Abuse and Alcoholism.

Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 48, 1373–1379.

Premack, D. (1965). Reinforcement theory. In D.Levine (Ed.), Nebraska Symposium on Motivation (Vol. 13). Lincoln: University of Nebraska Press.

Rachlin, H. (1971). On the tautology of the matching law. Journal of the Experimental Analysis of Behavior, 15, 249–251.

Rachlin, H. (1995). Self-control: Beyond commitment. Behavioral and Brain Sciences, 18, 109–159.

Rachlin, H., Battalio, R., Kagel, J., & Green, L. (1981). Maximization theory in behavioral psychology. Behavioral and Brain Sciences, 4, 371–417.

Reynolds, B., Richards, J. B., Horn, K., & Karraker, K. (2004). Delay discounting and probability discounting as related to cigarette smoking status in adults. Behavioural Processes, 65, 35–42.

Rosenberg, H. (2004). International research—Target groups. In H.Klingemann, R.Room, H.Rosenberg, S.Schatzmann, L.Sobell, & M.Sobell (Eds.), Kontrolliertes Trinken als Behandlungsziel—Bestandesaufnahme des aktuellen Wissens [Controlled drinking: A review] (Research report F03–90, pp. 69–79). Bern, Switzerland: University of Applied Sciences, School of Social Work.

Saladin, M. E., & Santa Ana, E. J. (2004). Controlled drinking: More than just a controversy. Current Opinions in Psychiatry, 17, 175–187.

Schmitz, J. M., Sayre, S. L., Hokanson, P. S., & Spiga, R. (2003). Assessment of the relative reinforcement value of smoking and drinking using a multiple-choice measurement strategy. Nicotine & Tobacco Research, 5, 729–734.

Selzer, M. L. (1971). The Michigan Alcoholism Screening Test: The quest for a new diagnostic instrument. American Journal of Psychiatry, 127, 1653–1658.

Skinner, H. A., & Horn, J. L. (1984). Alcohol Dependence Scale (ADS) user's guide. Toronto, Ontario, Canada: Addiction Research Foundation.

Sobell, L. C., & Sobell, M. B. (1992). Timeline followback: A technique for assessing self-reported alcohol consumption. In R.Litten & J.Allen (Eds.), Measuring alcohol consumption (pp. 41–72). Totowa, NJ: Humana Press.

Sobell, L. C., Sobell, M. B., & Toneatto, T. (1992). Recovery from alcohol problem without treatment. In N.Heather, W. R.Miller, & J.Greeley (Eds.), Self-control and addictive behaviors (pp. 198–242). New York: Macmillan.

Tucker, J. A. (2003). Natural resolution of alcohol-related problems. In M.Galanter & J. P.Allen (Eds.), Recent developments in alcoholism: Vol. 16. Research in alcoholism treatment (pp. 77–90). New York: Plenum Press.

Tucker, J. A., Foushee, H. R., & Black, B. C. (2008). Behavioral economics of natural resolution of drinking problems using IVR self-monitoring. Experimental and Clinical Psychopharmacology, 16, 332–340.

Tucker, J. A., Foushee, H. R., Black, B. C., & Roth, D. L. (2007). Agreement between prospective IVR self-monitoring and structured retrospective reports of drinking and contextual variables during natural resolution attempts. Journal of Studies on Alcohol and Drugs, 68, 538–542.

Tucker, J. A., Vuchinich, R. E., Black, B. C., & Rippens, P. D. (2006). Significance of a behavioral economic index of reward value in predicting problem drinking resolutions. Journal of Consulting and Clinical Psychology, 74, 317–326.

Tucker, J. A., Vuchinich, R. E., & Rippens, P. D. (2002). Predicting natural resolution of alcohol-related problems: A prospective behavioral economic analysis. Experimental and Clinical Psychopharmacology, 10, 248–257.

Vuchinich, R. E., & Tucker, J. A. (1996). The molar context of alcohol abuse. In L.Green & J. H.Kagel (Eds.), Advances in behavioral economics: Vol.3. Substance use and abuse (pp. 133–162). Norwood, NJ: Ablex.

Vuchinich, R. E., & Tucker, J. A. (1998). Choice, behavioral economics, and addictive behavior patterns. In W. R.Miller & N.Heather (Eds.), Treating addictive behaviors: Processes of change (2nd ed., pp. 93–104). New York: Plenum.

Vuchinich, R. E., Tucker, J. A., & Harllee, L. M. (1988). Behavioral assessment. In D. M.Donovan & G. A.Marlatt (Eds.), Assessment of addictive behaviors (pp. 203–223). New York: Wiley.

World Health Organization. (2000). International guide for monitoring alcohol consumption and related harm (WHO/MSD/MSB/00.4). Geneva: World Health Organization, Noncommunicable Diseases and Mental Health Cluster, Department of Mental Health and Substance Dependence.

Yoon, J. H., Higgins, S. T., Heil, S. H., Sugarbaker, R. J., Thomas, C. S., & Badger, G. J. (2007). Delay discounting predicts postpartum relapse to cigarette smoking among pregnant women. Experimental and Clinical Psychopharmacology, 15, 176–186.

Submitted: December 27, 2007 Revised: December 5, 2008 Accepted: December 16, 2008

A Communication-Based Intervention for Nonverbal Children With Autism: What Changes? Who Benefits?

Acknowledgement: We thank all the participating schools, children, and their parents; PECS Consultants Sue Baker and Teresa Webb from Pyramid UK; and The Three Guineas Trust for their generous support of this project.

Over the past two decades, there has been accumulating evidence for the effectiveness of psychosocial programs for young children with autism. These tend to incorporate a mix of behavioral, developmental, and educational approaches, and although methods may vary, their general goals are to enhance cognitive, communication, and social skills while minimizing rigid and repetitive and other problem behaviors (see Lord & McGee, 2001; Rogers & Vismara, 2008, for reviews). In part due to the design of studies, however, there have been few opportunities to examine the detail of exactly what changes and who benefits as a result of these interventions.

In terms of measuring what changes, many early autism intervention studies, in particular within the applied behavior analysis field, used global measures of outcome such as IQ scores and school placement (Dawson & Osterling, 1997; Howlin, Magiati, & Charman, 2009). More recent studies have used a wider range of measures, including standardized tests of adaptive behavior, expressive and receptive language, and measures of autism severity (Howard, Sparkman, Cohen, Green, & Stanislaw, 2005; Remington et al., 2007). Some studies have attempted to include measures of change in skills or behaviors specifically targeted by the intervention, for example, parents' or carers' knowledge about autism (Jocelyn, Casiro, Beattie, Bow, & Kneisz, 1998). Furthermore, a small number of studies include naturalistic or quasinaturalistic measures of communication, for example, observing spontaneous communication and language in parent–child interactions (Aldred, Green, & Adams, 2004); observing parents' use of facilitative strategies during social interaction with their child (McConachie, Randle, Hammal, & Le Couteur, 2005); recording the frequency and rate of children's turn-taking, joint attention, and requesting behaviors (Yoder & Stone, 2006a, 2006b); and observing structured play and joint attention acts within parent– and experimenter–child interactions (Kasari, Freeman, & Paparella, 2006; Kasari, Paparella, Freeman, & Jahromi, 2008). In addition to increasing the face validity of the research, demonstration of change in specific behaviors is likely to be helpful in elucidating exactly how an intervention is working (Kazdin & Nock, 2003).

To examine who benefits from intervention, attempts have been made to study the effects of preintervention child characteristics on outcome, most notably IQ and age. For IQ, this usually involves examining correlations between preintervention IQ and postintervention outcome (Eldevik, Eikeseth, Jahr, & Smith, 2006; Harris & Handleman, 2000; Remington et al., 2007) or the comparison of outcomes for high-IQ versus low-IQ subgroups (Aldred et al., 2004; Ben Itzchak, Lahat, Burgin, & Zachor, 2008; Ben-Itzchak & Zachor, 2007). Significant positive associations have been found between preintervention IQ and outcome. However, Yoder and Compton (2004) highlighted the flaws of testing for moderators by exploring correlations or comparison of subgroups' effect sizes. They emphasized the importance of using statistical methods that enable differentiation of predictors of growth or progress from predictors of intervention response. Where participants have been randomly assigned to intervention or control conditions, the appropriate method for identifying predictors of intervention response is to test for statistical interactions between child characteristics and group assignment in relation to the outcome variables. Although such statistical tests of moderator effects are well established in medical trial literature and used increasingly in the psychiatry field (Kraemer, Wilson, Fairburn, & Agras, 2002) to date, they have only been used in two studies of autism intervention (Kasari et al., 2006, 2008; Yoder & Stone, 2006a, 2006b).

Kasari et al. (2006) randomized 58 preschool children with autism (aged 3–4) to either joint attention training, symbolic play training, or a control condition, demonstrating that both active interventions were effective at enhancing social communication skills. In a later study, they presented further analysis revealing that growth in expressive language was positively predicted by a number of joint attention and symbolic play variables (Kasari et al., 2008). Yoder and Stone (2006a, 2006b) used multiple regression to demonstrate the moderating effects of baseline joint attention abilities. Their randomized trial (N = 36) compared Picture Exchange Communication Training (PECS; Bondy & Frost, 1998) with Responsive Education and Prelinguistic Milieu Training (RPMT; Yoder & Warren, 1999). Children who initiated joint attention relatively more frequently at baseline benefited more from RPMT in terms of their postintervention frequency of joint attention initiations, whereas children who initiated joint attention less frequently at baseline benefited more from PECS (Yoder & Stone, 2006a). In a later analysis, using mixed-level modeling, they found object exploration also moderated intervention response (Yoder & Stone, 2006b). Thus, children who displayed object exploration behaviors more frequently at baseline benefited most from PECS, showing greater increases in production of nonimitative words at outcome. The children who showed lower object exploration at baseline benefited more with respect to word production if they had received RPMT.

These studies notwithstanding, to date relatively few studies of psychosocial intervention for children with autism have effectively investigated what changes in response to treatment and who benefits. Common use of standardized IQ and language assessments to measure outcome has meant there has been relatively little naturalistic analysis of change in the form and function of children's communication as a result of intervention. Insufficiently sized samples and lack of randomization have impeded investigation into treatment moderators.

Enhancing spontaneity in everyday communication has been highlighted as one of the most important goals of intervention for children with autism (Lord & McGee, 2001). The aim of PECS is to teach spontaneous and functional communication to children with autism in a social context (Bondy & Frost, 1998). PECS is widely used in home and educational settings (Preston & Carter, 2009; Sulzer-Azaroff, Hoffman, Horton, Bondy, & Frost, 2009), and there is evidence from a number of single-case and case-series studies (Charlop-Christy, Carpenter, Le, LeBlanc, & Kellet, 2002; Ganz & Simpson, 2004; Kravits, Kamps, Kemmerer, & Potucek, 2002; Magiati & Howlin, 2003; Schwartz, Garfinkle, & Bauer, 1998), a school-based controlled study (Carr & Felce, 2007), and two randomized controlled trials (RCTs) (Howlin, Gordon, Pasco, Wade, & Charman, 2007; Yoder & Stone, 2006a, 2006b) that PECS can lead to improved communication skills in this group.

Howlin et al. (2007) conducted a pragmatic RCT of PECS. The study was designed to test the “real world” effectiveness of PECS. Initial training was delivered to teachers and classroom assistants by PECS consultants at workshops. PECS was subsequently applied by school staff in the classroom under regular supervision by PECS consultants. Naturalistic observations of rates of children's communication were used as the primary outcomes to measure intervention effects. Immediately after training had ended (approximately 5 months), the rate at which children spontaneously initiated communication (IC) had significantly increased. Overall rates of children's use of picture cards (P) to communicate (i.e., spontaneous or prompted) had also significantly increased. By the 9-month follow-up, however, these effects had disappeared. Overall rates of children's speech/vocalization (S) (including spontaneous and prompted) did not increase.

In the present article, using the sample from Howlin et al. (2007), we explore exactly which communication forms were used by children more spontaneously as a result of PECS training, which communicative functions increased and which children benefited most from the intervention. We aimed to build on previous studies by applying appropriate analysis to this relatively large sample to address the following questions:

1. Did PECS training act specifically to increase children's spontaneous communication using the picture cards or did its effect generalize to support greater spontaneity using speech as well? Previous studies have demonstrated that PECS supports children to communicate spontaneously using picture cards (e.g., Bondy & Frost, 1994; Charlop-Christy et al., 2002; Ganz & Simpson, 2004; Kravits et al., 2002) and sometimes speech (e.g., Bondy & Frost, 1994; Charlop-Christy et al., 2002; Ganz & Simpson, 2004). Two studies using more naturalistic measures of outcome also suggest that PECS can increase spontaneous communication using picture cards and speech together (Carr & Felce, 2007; Yoder & Stone, 2006a).

2. Did PECS increase children's spontaneous communication purely for instrumental requesting or did the training also lead to increased spontaneous communication for more social purposes? PECS training initially focuses on teaching children to make requests for objects. Later training phases aim to broaden the range of communicative functions, such as sharing attention through commenting (Bondy, Tincani, & Frost, 2004). Studies have shown that PECS training can be used successfully to teach children spontaneous requesting for objects (Carr & Felce, 2007; Ganz & Simpson, 2004; Kravits et al., 2002), and some have demonstrated effects on other forms of noninstrumental, more social communication (Schwartz et al., 1998; Yoder & Stone, 2006b).

3. Which children benefited most from PECS training? PECS was specifically developed for children with autism to obviate the need for prerequisite communication skills. It might be hypothesized therefore that response to PECS training would not be predicted by such factors as language comprehension skills or autistic symptomatology. Within the autism intervention literature more generally, however, higher IQ has been associated with better outcome (e.g., Harris & Handleman, 2000; Schwartz et al., 1998). Yoder and Stone's (2006a, 2006b) studies have been the only systematic investigation of PECS response predictors to date, finding that children who were most impaired in terms of baseline language and joint attention skills were those who gained most from PECS training. Given the lack of research in this area, we took an exploratory approach in the present study, investigating the potential moderating effect on PECS training of four factors measured at baseline: chronological age, expressive language, autistic symptomatology, and cognitive ability.

Method

Participants of the RCT

Eighty-four children (73 boys, 11 girls) from 17 classes in special needs elementary schools participated in the study. Classes were broadly similar, with a child–adult ratio of approximately 2:1. Class teaching programs varied, but most classes adopted an eclectic approach incorporating a range of visual systems and structured teaching, often based on the TEACCH methodology (Mesibov, Shea, & Schopler, 2004). Picture cards were present in most classrooms in the treatment and nontreatment groups even at baseline, though these were not necessarily used according to PECS principles. All schools were situated in Greater London or the south east of England. Children were aged between 4 and 10 years (mean age at baseline = 6.8 years, SD = 1.26), and all had an intellectual disability. Ethnicity, socioeconomic status, and comorbidity data were not formally collected. This was a community-based study that included all suitable children whose parents consented. To be eligible, children had to (a) have a formal clinical diagnosis of autism, (b) use little or no functional language (i.e., no more than single words), (c) have no sensory impairment, and (d) be aged between 4 and 11 years and (e) not using PECS beyond Phase 1.

Informed consent and ethical approval

Written informed consent to participate in the study was obtained from the parent or guardian of each child and from a senior member of staff from each school. The original trial protocol was prospectively reviewed and approved by the Wandsworth Local Research Ethics Committee (Ref. IAS/der/02.42.6).

Describing the group at baseline

To obtain baseline data on autism severity, all children were assessed using Module 1 of the Autism Diagnosis Observation Schedule (ADOS; Lord et al., 2000). The ADOS is an interactive semistructured assessment of communication, social interaction, imagination, and repetitive and stereotyped interests. Assessment consists of a range of activities and social presses providing a standardized context in which to observe specific behaviors. There are four modules. Module 1 was used in this study, designed for use with preverbal individuals or for those whose expressive language is still at single word or simple phrase level. Seventy-five children met the ADOS criteria for a diagnosis of autism; nine children met criteria for autism spectrum disorder. Score from Item A1 of the ADOS was used as an index of expressive language ability (0 = regular use of utterance of two or more words; 1 = occasional phrases only, mostly single words; 2 = recognizable single words only; 3 = at least one word or word approximation but fewer than five words; 4 = no words or word approximations). Thirty-eight children (45%) used no words or word approximations during the ADOS, 31 (37%) used single words, and 15 (18%) used at least one phrase. Most children (64%) scored 0 on the Expressive One Word Picture Vocabulary Test (Brownell, 2000). Nonverbal developmental quotient (NVDQ = nonverbal mental age equivalent/chronological age ×100) was ascertained using the Mullen Scales of Early Learning (Mullen, 1995). Group median NVDQ was 29.90 (interquartile range was 21.20–40.52). In summary, the sample comprised children with clear autism and who were very impaired with regard to verbal and nonverbal skills.

Design of the RCT

As a group-randomized control trial, class groups (each including approximately six children and two to three staff) were assigned into one of three intervention groups. The immediate treatment group (ITG; five class groups, 26 children) received training immediately after the baseline assessment; the delayed treatment group (DTG; six class groups, 30 children) received training about 9 months later, immediately after Time 2 assessment; the no-treatment group (NTG; six class groups, 28 children) received no training. Staggering of treatment across two time periods maximized the number of children involved in the study and allowed investigation of the continued effectiveness of any immediate treatment effects noted. The data analyses incorporated each child contributing all measurements within all control, treatment, and posttreatment periods; thus, statistical power was not compromised by the three-arm approach to data collection. Differences between the three groups at baseline were analyzed and reported in Howlin et al. (2007). The analysis was designed to adjust for these differences. Figure 1 shows recruitment, the points at which intervention was delivered, and when each of the three groups was observed.

Figure 1. Flow chart illustrating sample selection, recruitment, training, and outcome assessment. Adapted from “The Effectiveness of Picture Exchange Communication System (PECS) Training for Teachers of Children With Autism: A Pragmatic, Group Randomised Controlled Trial,” by P. Howlin, R. Kate Gordon, G. Pasco, A. Wade, and T Charman, 2007, Journal of Child Psychology and Psychiatry, 48, pp. 473–481. Copyright 2007 by John Wiley and Sons.

Outcome measurement

The outcome measure was a 15-min videotaped observation, intended to be an ecologically valid measure of communication skills. Children were filmed in their class snack sessions at Time 1 (baseline) and twice further over a period of 20 months (2 academic years). Snack sessions were selected as these were likely to create the most opportunities for children to make spontaneous requests. Furthermore, daily snack sessions occurred in all the classes and were broadly similar. These sessions usually lasted approximately 15 min and involved all children and class staff sitting at tables in the classroom or school kitchen. Drinks and food snacks such as fruit or cookies were given out or were on offer for children to request. Where classes used picture cards, these were usually made available for children (e.g., by placing a large board at the front of the classroom or by handing out books with the cards inside).

Children's communication was coded from the videotape using an observation schedule designed specifically for this study (Classroom Observation Schedule for Measuring Intentional Communication; Pasco, Gordon, Howlin, & Charman, 2008). The primary outcome variable was frequency of child-initiated communication (IC). Frequencies of different communication modalities used (such as the number of times a child used a picture card [P] and/or speech/vocalization [S] to communicate) were also recorded; communication functions were recorded by counting each time a child communicated for the purpose of requesting objects (R) and for the purpose of requesting a social interaction or commenting (D). In this way, a single communication act might produce three or more codes, for example, as a spontaneous initiation (IC), of the use of a picture card (P), and for the purpose of requesting (R).

Data analysis

Where outcomes are numerical counts of relatively rare events, Poisson regression is a useful method for analysis (Dobson, 2002). The Poisson regression model expresses the log outcome rate as a linear function of a set of predictors. In this study, Poisson regression models were produced for each of the five outcome variables of interest, concerned with form or function of children's spontaneously initiated communication: spontaneous communicative initiation using picture cards (IC-P); spontaneous communication using speech (IC-S); spontaneous communication using both simultaneously (IC-PS); spontaneous communication to request for objects (IC-R), and spontaneous communication to request for social routine or commenting (IC-D). The regression models were created within the Stata IC Version 10 (StataCorp., 2003).

As can be seen in Figure 1, the data set comprised data from three time points in the three different experimental groups. Multilevel models were used that took account of the longitudinal nature of the measurements, time trends, differing treatment regimes within the same individuals over time, and within child correlations between repeated measurements (Goldstein, 2003). The standard errors of the model parameters were thus adjusted for any within-child (across time) or within-class (between children) correlations. Models also allowed adjustment for any group differences at baseline in terms of age, developmental level, expressive language, and autistic symptom severity. Each model included an independent binary intervention variable (i.e., intervention or no intervention), a further binary variable to denote follow-up (this occurred at Time 3 for the ITG group only), a time variable (continuous in order to adjust for differences in the actual lengths of time between observations, i.e., Time 1 = 0 days), and an offset to adjust for the difference in the lengths of snack times for individual children.

In Poisson regression, effect size is represented by the rate ratio (RR) that estimates the relative rate of change in the mean number of events attributable to each explanatory variable. For example, for a binary intervention variable, the RR represents the relative difference in mean frequency of spontaneous initiations for children in the intervention group compared with those not in the intervention group. For continuous variables (e.g., baseline age [months]), the RR represents the relative difference in the mean frequency of initiations for every increase in one unit of the explanatory variable—in this case, for every month older the child was at baseline. An RR of 1 indicates no change, and so, for example, an RR of 1.2 represents an increase of 20% for each unit increase; an RR of 0.7 represents a decrease of 30% for each unit increase. RRs for estimates from the five models (for each of the five outcome variables) are reported along with 95% confidence intervals and p values. These models were not independent and were interpreted jointly taking into account the relationship between the various outcomes.

Testing for intervention moderators

Where PECS had a significant effect, a second round of analyses was conducted in order to identify potential intervention response moderators. If baseline factors (i.e., chronological age, autistic symptomatology, expressive language, or developmental quotient) independently predicted progress at postintervention (shown in Table 2), tests for Intervention × Baseline Factor interactions were conducted to explore whether they also predicted a specific intervention response. The RR for the interaction term represents the impact of the baseline variable on the outcome over and above any existing variance due to a main intervention effect or a main effect of the baseline factor.

Rate Ratio Estimates (and 95% Confidence Intervals) for Each of the Five Outcome Variables

Results

The results are presented in two parts. First, we examined the impact of PECS training on children's spontaneous communication using three different communication modalities and for two different functions. Second, controlling for differences, we tested baseline variables for their potential moderating effect on the intervention.

Change in spontaneous communication following PECS training

Table 1 shows the median rate of initiations per minute for the five variables in each of the three treatment arms at each of the three time points. The rates in bold are immediately following PECS training in the ITG and DTG. The italicized rates are at 9-month follow-up (ITG only). These figures indicate some changes following PECS training. For example, in the DTG, the median rate of spontaneous initiation of communication using picture cards went from 0 to 0.44 per minute, that is, more than 6 times per 15-min snack session. In the ITG, the median rate of spontaneous communication using speech or vocalization went from 0.03 to 0.13 per minute, and in the DTG, the median rate of spontaneous requesting rose from 0.03 to 0.46 times per minute. Despite these group effects, for each of the form and function variables, some children remained at zero, even after PECS training. For example, of the 56 children in the ITG and DTG, 12 were still not using picture cards to spontaneously communicate at all after the training, and nine were still not making spontaneous requests.

Median Initiations Made Per Minute at Each Time Point for Each of the Three Treatment Groups

Table 2 shows the results of the Poisson analysis for each variable. RRs are shown for change attributable to the intervention immediately post-PECS training, at 9-month follow-up (for the ITG group only), and for each of the nonintervention variables measured at baseline. Initiations using picture cards (IC-P), using speech (IC-S), and using both simultaneously (IC-PS) all increased significantly following training (RR = 1.90, 95% CI [1.46, 2.48], p < .001; RR = 1.77, 95% CI [1.35, 2.32], p < .001; RR = 3.74, 95% CI [2.19, 6.37], p < .001, respectively). The average increase observed was similar in size for IC-P and IC-S and about twice as large for IC-PS. However, it should be noted that the confidence intervals are wide, and in all instances the data are compatible with a twofold increase in the RR, so we cannot necessarily infer that the effect is any greater for IC-PS. Spontaneous requesting for objects (IC-R) significantly increased following training (RR = 2.17, 95% CI [1.75, 2.68], p < .001), but requesting for social routine or commenting (IC-D) did not (RR = 1.34, 95% CI [0.83, 2.18], p = .237). Children in the ITG (n = 26) were observed again at follow-up (approximately 9 months after the end of the training period). Although the effect on spontaneous initiation using speech/vocalization (IC-S) had persisted (RR = 1.70, 95% CI [1.12, 2.58], p = .012), none of the other effects were significant (IC-P, RR = 0.69, 95% CI [0.41, 1.15], p = .15; IC-PS, RR = 1.90, 95% CI [0.76, 4.76], p = .17; IC-R, RR = 1.11, 95% CI [0.76, 1.62], p = .60).

Variables moderating the effect of PECS training

We analyzed baseline variables for their power to predict progress in general and to predict specific response to treatment. Seven baseline variables (shown in bold in Table 2) were independently and significantly related to general progress at postintervention and so were testable as potential moderators of the intervention effects. Of these seven Intervention × Baseline Variable interactions tested, two were found to be significant. The impact of the intervention on children's spontaneous initiation of communication using speech/vocalization (IC-S) was moderated by baseline autistic symptomatology (RR = 0.90, 95% CI [0.83, 0.98], p = .011). As can be seen in Figure 2, children whose autistic symptomatology score was lowest at baseline (i.e., least severe symptoms) showed the largest increases in spontaneous use of speech/vocalization following intervention. Each unit increase in ADOS score was associated with a 10% decrease in average rate of initiation using speech/vocalization (IC-S). Baseline expressive language did not moderate intervention effects for this outcome (RR = 1.05, 95% CI [0.90, 1.24], p = .524).

Figure 2. Graph showing the moderating effect of autistic symptomatology on the effect of PECS on children's spontaneous communication using speech/vocalization (Autism Diagnosis Observation Schedule [ADOS] scores are on a severity scale; higher score means more severe symptomatology).

Baseline expressive language moderated the effect of PECS training on children's spontaneous initiation using picture cards and speech/vocalization together (RR = 0.62, 95% CI [0.44, 0.88], p = .008). Expressive language was rated on a severity scale. Figure 3 shows that those children with the most expressive language at baseline (lower score represents better expressive language) showed the biggest increase in their use of picture cards and speech/vocalization together to spontaneously initiate communication. Each unit increase in expressive language deficit score was associated with a 38% decrease in the average rate of initiations using picture cards and speech together (IC-PS). Neither baseline developmental quotient nor autistic symptomatology moderated the effects of the intervention for this outcome (RR = 1.01, 95% CI [0.98, 1.05], p = .539 and RR = 0.94, 95% CI [0.79, 1.12], p = .477, respectively). As can be seen in Table 2, baseline developmental quotient (DQ) predicted of rate of initiation using picture card (IC-P) and rate of initiation for the purpose of instrumental requesting (IC-R) immediately postintervention, but interaction tests demonstrated that this did not moderate the effects of the training on these behaviors (for IC-P, DQ × Intervention, RR = 0.99, 95% CI [0.98, 1.01], p = .214; for IC-R, DQ × Intervention, RR = 0.99, 95% CI [0.98, 1.00], p = .212).

Figure 3. Graph showing the moderating effect of baseline expressive language on children's spontaneous communication using picture cards and speech/vocalization. ADOS = Autism Diagnosis Observation Schedule.

Discussion

PECS is recognized as an effective intervention for increasing communication in children with autism (Preston & Carter, 2009; Sulzer-Azaroff et al., 2009), and our RCT demonstrated specifically that PECS training can significantly enhance the spontaneity of children's communication (Howlin et al., 2007). In this article, we asked exactly how PECS training increased this communicative spontaneity and for which children. That is, we wanted to examine, first, whether the increased spontaneity was confined to communication using the picture symbols or whether PECS also impacted on the spontaneity of children's use of speech/vocalization. Second, we wished to examine whether the increased spontaneous communication was being used only for instrumental purposes (e.g., getting a snack) or whether children were also spontaneously initiating communication for more social purposes as a result of PECS training. Third, we wanted to identify factors that might be moderating the effect of PECS training and therefore predictive of which children might benefit most from the training. We used Poisson regression analysis to examine the children's spontaneous communication using different communication modalities and for different functions and to test for interactions between the intervention and baseline child variables.

The naturalistic and relatively fine-grained outcome measurement meant that it was possible to analyze exactly how PECS was enhancing children's spontaneous communication in an everyday situation. A small number of previous intervention studies have examined the form of children's communication but have not focused purely on spontaneous unprompted communication. The present analyses revealed that although PECS training did lead to children spontaneously communicating more using the picture cards, it also led to increased spontaneity in children's use of speech and their use of picture cards and speech in combination. The training appears to have increased spontaneous requesting for objects or help but not spontaneous requesting for social routine or commenting.

In contrast to some other reports (Bondy & Frost, 1994; Charlop-Christy et al., 2002), in our primary analysis of PECS RCT, we did not observe an effect of the intervention on overall use of speech (Howlin et al., 2007). The present analysis revealed, however, that PECS did enhance the use of speech as a modality to spontaneously initiate communication, as well as enhancing spontaneity using picture cards. So, although it would appear that PECS training did not enhance speech development per se, for those children who were already using some speech or vocalization, PECS appears to have provided a structure for them to use this mode to communicate without prompting. It would seem that PECS fostered spontaneity more generally across modalities rather than just acting to increase children's use of picture cards. Furthermore, the effect of PECS training on children's spontaneous speech/vocalization appears to have been particularly robust as it was also observed 9 months after the end of the training period in the group who received PECS training early on. There was no long-term effect on spontaneous use of picture cards.

Detailed analysis of the functions of spontaneous communication in autism intervention is also rare. In this study, analysis revealed a clear effect on children's spontaneous communication for the purposes of requesting for objects, such as a drink or a toy, which is the first communicative function taught through PECS teaching phases (Frost & Bondy, 2002). This replicates findings from earlier research (Schwartz et al., 1998; Yoder & Stone, 2006a, 2006b). There was no effect of training on children's spontaneous communication for social purposes. This might be due to the fact that the children in this sample had severe autism symptoms and, as a group, were very delayed with regard to verbal and nonverbal skills. Furthermore, the discrepancy between instrumental and social communication is perhaps to be expected given that the children were observed in class snack sessions. It is possible that observation of children in other nonsnack sessions might have revealed effects of training on communication for other noninstrumental purposes. Also, it is possible that if the training had persisted for longer or had been more intense, changes in spontaneous social, noninstrumental communication might have been seen. Some case study reports have described children successfully learning to communicate for social interaction purposes such as commenting (e.g., Schwartz et al., 1998; Webb, 2000), although, to date, no experimental trials have demonstrated this effect of PECS.

Two baseline variables appeared to moderate the effect of PECS training. First, less severe autistic symptomatology at baseline predicted the greatest increases in spontaneous speech. Second, higher level of expressive language at baseline predicted greater increases in spontaneous use of speech and picture cards together. This is to be expected, as more severe autism and greater language disability are not independent. Thus, the fact that the least severely autistic children and those with the most expressive language showed the greatest improvements in these areas is consistent with the autism intervention literature more generally (e.g., Harris & Handleman, 2000; Kasari et al., 2008). We observed no interactions between PECS training and any of the abilities measured at baseline on children's spontaneous use of the picture cards or spontaneous requesting. Yoder and Stone's (2006a,2006b) studies have been the only other systematic examination of moderators of PECS intervention. They compared PECS with RPMT, and although there was no overall difference between the interventions, children who were most impaired in baseline language and joint attention skills gained most in terms of their joint attention skills from PECS training, whereas the more able children made better progress with RPMT. The present study did not replicate the finding that the less able children benefited more from PECS. A potential explanation is that, as a group, the children in the present sample were less able than Yoder and Stone's sample. In Yoder and Stone's sample, the mean nonverbal mental age was 18.8 months (standard deviation 4.5 months) at 3 years of age (Table 1 in Yoder & Stone, 2006a, p. 429), meaning that the mean NVDQ was approximately 50, whereas the mean NVDQ in our sample was around 30.

Despite the fact that all children in the present study were very impaired in terms of their verbal and nonverbal skills, spontaneous use of pictures to communicate and spontaneous requesting did increase, and this was not predicted by better baseline language or less severe autism symptoms. This suggests that PECS training was equally accessible to these children in terms of teaching these skills specifically. This seems to support the idea that, beyond the need for some very basic cognitive skills required in order to exchange the cards (e.g., object permanence), few preexisting verbal or nonverbal skills are required to learn to use PECS (Bondy & Frost, 1998).

Strengths and limitations of the present study

The unique quality of the data presented here is that they are derived from an examination of “real world” effectiveness. The study took an inclusive approach to recruitment, aiming to include all suitable schools within a defined but large geographical area in the south east of England and included all suitable children whose parents consented. As the trial was community based, intervention was delivered to teachers and classroom staff via a workshop and follow-up visits to the schools. Teachers had to implement the program amidst all the other pressures and distractions of running a classroom for children with special educational needs and from a wide variety of backgrounds. Children were required to access the intervention in spite of their severe autistic symptoms, language impairments, and perhaps other comorbid problems. In other words, PECS training that was delivered and evaluated seemed to be a realistic representation of PECS training most children are likely to receive.

The design of the study and analysis ensured that the use of three treatment groups did not detract from the numbers effectively used for the intervention and no-intervention groups. The study was, in fact, strengthened by having within-individual comparisons (i.e., the delayed intervention group) as well as between-group comparisons over the same time frame. The use of the multilevel model allowed for efficient use of data in the three-treatment arm format adopted in this study, taking into account correlations between repeats from the same individuals and allowing for the serial nature of measurements under different treatment conditions. The design also enhanced the power to investigate the immediate effects of the intervention and enabled investigation of the longer term effects of the training. The incorporation of baseline data further strengthened the results.

The interaction analysis applied in this study is relatively novel to this field and demonstrates the possibility of using relatively sophisticated statistical models to test for moderator effects on interventions for children with autism. As has been discussed above, this has been rarely done in the autism field in the past, and thus there is very little reliable information about who benefits most from various interventions for children with autism.

As this trial was conducted primarily in schools, we had little direct contact with parents, aside from their consenting for their child to take part in the study. As a consequence, we did not collect systematic information on family variables such as ethnicity and other background factors (socioeconomic status, parental income) that might also be related to differential outcome, nor did we collect detailed information on potential school moderating factors. Instead, we focused on child characteristics as moderating factors. Generalization of these findings will require replication in samples with well-described demographic information as well as well-characterized schools/classrooms.

For logistical reasons (i.e., limited resources), we were limited to observing children in their classrooms. We opted to observe them during snack sessions as this was a session that created more opportunities for children to make spontaneous requests, relative to less structured sessions, and this was a common feature on the timetables of all classes involved in the study. However, the snack sessions are relatively brief periods, when children are usually highly motivated to make approaches for food, and so the data may not represent changes in children's communication in other less structured or less motivating contexts. Observations of children communicating in other class sessions or at home would have revealed the extent to which the observed effects generalized out of the relatively structured setting of class snack time.

With regard to the analysis of intervention response predictors, we were limited to testing four child factors measured at baseline. It is possible that other factors, not measured, were moderating the intervention effects, including those that were external to the children (i.e., environmental factors). It is likely that differences between the classes and the ways in which PECS was implemented also influenced children's progress. Treatment fidelity measures will be important for future studies of psychosocial interventions.

A limitation of the analysis of response predictors was that, although relatively sophisticated, essentially it was based on comparing subgroups, and the study was not primarily powered for this. The chance of Type II errors is thus increased. The results do, however, provide a good basis for further discussion. In the future, as more intervention studies are conducted and there is greater consistency in approach across research, pooling of samples may be possible, thus increasing statistical power for identifying invention response moderators.

Implications

In summary, the findings show that classroom-based PECS training enhances children's ability to make spontaneous instrumental requests not only using pictures but also using speech, or a combination of both. It also shows that, similar to other interventions, less impaired children appear to show the most improvement in these areas. Where these improvements were seen, they represented noticeable change in children's communication. For example, in one treatment group, the median rate of spontaneously initiated communication using PECS went from 0 times per 15-min snack session up to more than 6 times, and the median rate of spontaneous requesting rose from ~0.5 times to ~7 times per snack session. It is important to remember, however, that these figures are based on group effects. Such impressive gains were not seen in all children who received PECS training, and for some children, no gains were made at all. Nevertheless, for a child who has not been communicating at all to request even twice in a 15-min snack session represents a meaningful change.

The study also has important methodological implications. This article builds on the findings of one of the larger RCTs conducted in the autism field to date (Howlin et al., 2007; though see Aman et al., 2009; Green et al., 2010, for larger studies). The adequately sized sample, well described in terms of verbal and nonverbal abilities, provides the opportunity for analysis of the specific effects of psychosocial intervention for children with autism. The study demonstrates the feasibility of applying robust statistical techniques to pragmatic, “real world” trials in this field to elucidate exactly what changes and for whom.

Aldred, C., Green, J., & Adams, C. (2004). A new social communication intervention for children with autism: Pilot randomised controlled treatment study suggesting effectiveness. Journal of Child Psychology and Psychiatry, 45, 1420–1430. doi:10.1111/j.1469-7610.2004.00338.x

Aman, M. G., McDougle, C. J., Scahill, L., Handen, B., Arnold, L. E., Johnson, C., . . .Wagner, A. (2009). Medication and parent training in children with pervasive developmental disorders and serious behavior problems: Results from a randomized clinical trial. Journal of the American Academy of Child & Adolescent Psychiatry, 48, 1143–1154. doi:10.1097/CHI.0b013e3181bfd669

Ben Itzchak, E., Lahat, E., Burgin, R., & Zachor, A. D. (2008). Cognitive, behavior and intervention outcome in young children with autism. Research in Developmental Disabilities, 29, 447–458. doi:10.1016/j.ridd.2007.08.003

Ben-Itzchak, E., & Zachor, D. A. (2007). The effects of intellectual functioning and autism severity on outcome of early behavioral intervention for children with autism. Research in Developmental Disabilities, 28, 287–303. doi:10.1016/j.ridd.2006.03.002

Bondy, A., & Frost, L. A. (1994). The Delaware Autistic Program. In S. L.Harris & J. S.Handleman (Eds.), Preschool education programs for children with autism (pp. 37–54). Austin, TX: Pro-Ed.

Bondy, A., & Frost, L. A. (1998). The Picture Exchange Communication System. Seminars in Speech and Language, 19, 373–389. doi:10.1055/s-2008-1064055

Bondy, A., Tincani, M., & Frost, L. (2004). Multiply controlled verbal operants: An analysis and extension to the picture exchange communication system. Behavior Analyst, 27, 247–261.

Brownell, R. (2000). Expressive One Word Picture Vocabulary Test. Novato, CA: Academic Therapy Publications.

Carr, D., & Felce, J. (2007). The effects of PECS teaching to phase III on the communicative interactions between children with autism and their teachers. Journal of Autism and Developmental Disorders, 37, 724–737. doi:10.1007/s10803-006-0203-1

Charlop-Christy, M. H., Carpenter, M., Le, L., LeBlanc, L. A., & Kellet, K. (2002). Using the picture exchange communication system (PECS) with children with autism: assessment of PECS acquisition, speech, social-communicative behavior, and problem behavior. Journal of Applied Behavior Analysis, 35, 213–231. doi:10.1901/jaba.2002.35-213

Dawson, G., & Osterling, J. (1997). Early intervention in autism: Effectiveness and common elements of current approaches. In M.Guralnik (Ed.), The effectiveness of early intervention: Second generation research (pp. 307–326). Baltimore, MD: Brookes Publishing.

Dobson, A. J. (2002). An introduction to generalized linear models (2nd ed.). London, England: Chapman & Hall.

Eldevik, S., Eikeseth, S., Jahr, E., & Smith, T. (2006). Effects of low-intensity behavioral treatment for children with autism and mental retardation. Journal of Autism and Developmental Disorders, 36, 211–224. doi:10.1007/s10803-005-0058-x

Frost, L., & Bondy, A. (2002). PECS: Picture Exchange Communication System training manual (2nd ed.). Cherry Hill, NJ: Pyramid Educational Consultants.

Ganz, J. B., & Simpson, R. L. (2004). Effects on communicative requesting and speech development of the Picture Exchange Communication System in children with characteristics of autism. Journal of Autism and Developmental Disorders, 34, 395–409. doi:10.1023/B:JADD.0000037416.59095.d7

Goldstein, H. (2003). Multilevel statistical models: Kendall's library of statistics3(3rd ed.). London, England: Hodder Arnold.

Green, J., Charman, T., McConachie, H., Aldred, C., Slonims, V., Howlin, P., . . .PACT Consortium. (2010). Parent-mediated communication-focused treatment in children with autism (PACT): a randomised controlled trial. The Lancet, 375, 2152–2160. doi:10.1016/S0140-6736(10)60587-9

Harris, S. L., & Handleman, J. S. (2000). Age and IQ at intake as predictors of placement for young children with autism: A four- to six-year follow-up. Journal of Autism and Developmental Disorders, 30, 137–142. doi:10.1023/A:1005459606120

Howard, J. S., Sparkman, C. R., Cohen, H. G., Green, G., & Stanislaw, H. (2005). A comparison of intensive behavior analytic and eclectic treatments for young children with autism. Research in Developmental Disabilities, 26, 359–383. doi:10.1016/j.ridd.2004.09.005

Howlin, P., Gordon, R. K., Pasco, G., Wade, A., & Charman, T. (2007). The effectiveness of Picture Exchange Communication System (PECS) training for teachers of children with autism: a pragmatic, group randomised controlled trial. Journal of Child Psychology and Psychiatry, 48, 473–481. doi:10.1111/j.1469-7610.2006.01707.x

Howlin, P., Magiati, I., & Charman, T. (2009). Systematic review of early intensive behavioral interventions for children with autism. American Journal on Intellectual and Developmental Disabilities, 114, 23–41.

Jocelyn, L. J., Casiro, O. G., Beattie, D., Bow, J., & Kneisz, J. (1998). Treatment of children with autism: A randomized controlled trial to evaluate a caregiver-based intervention program in community day-care centers. Journal of Developmental and Behavioral Pediatrics, 19, 326–334. doi:10.1097/00004703-199810000-00002

Kasari, C., Freeman, S., & Paparella, T. (2006). Joint attention and symbolic play in young children with autism: A randomized controlled intervention study. Journal of Child Psychology and Psychiatry, 47, 611–620. doi:10.1111/j.1469-7610.2005.01567.x

Kasari, C., Paparella, T., Freeman, S., & Jahromi, L. B. (2008). Language outcome in autism: Randomized comparison of joint attention and play interventions. Journal of Consulting and Clinical Psychology, 76, 125–137. doi:10.1037/0022-006X.76.1.125

Kazdin, A. E., & Nock, M. K. (2003). Delineating mechanisms of change in child and adolescent therapy: methodological issues and research recommendations. Journal of Child Psychology and Psychiatry and Allied Disciplines, 44, 1116–1129. doi:10.1111/1469-7610.00195

Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002). Mediators and moderators of treatment effects in randomized clinical trials. Archives of General Psychiatry, 59, 877–883. doi:10.1001/archpsyc.59.10.877

Kravits, T. R., Kamps, D. M., Kemmerer, K., & Potucek, J. (2002). Brief report: Increasing communication skills for an elementary-aged student with autism using the Picture Exchange Communication System. Journal of Autism and Developmental Disorders, 32, 225–230. doi:10.1023/A:1015457931788

Lord, C., & McGee, J. P. (2001). Educating children with autism. Committee on educational interventions for children with autism. Division of Behavioural and Social Sciences and Education. Washington, DC: National Academy Press.

Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., . . .Rutter, M. (2000). The Autism Diagnostic Observational Schedule—Generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30, 205–223. doi:10.1023/A:1005592401947

Magiati, I., & Howlin, P. (2003). A pilot evaluation study of the Picture Exchange Communication System (PECS) for children with autistic spectrum disorders. Autism, 7, 297–320. doi:10.1177/13623613030073006

McConachie, H., Randle, V., Hammal, D., & Le Couteur, A. (2005). A controlled trial of a training course for parents of children with suspected autism spectrum disorder. Journal of Pediatrics, 147, 335–340. doi:10.1016/j.jpeds.2005.03.056

Mesibov, G. B., Shea, V., & Schopler, E. (2004). The TEACCH approach to autism spectrum disorders. New York, NY: Plenum Press.

Mullen, E. M. (1995). Mullen Scales of Early Learning. Circle Pines, MN: American Guidance Service.

Pasco, G., Gordon, R. K., Howlin, P., & Charman, T. (2008). The Classroom Observation Schedule to Measure Intentional Communication (COSMIC): An observational measure of the intentional communication of children with autism in an unstructured classroom setting. Journal of Autism and Developmental Disorders, 38, 1807–1818. doi:10.1007/s10803-008-0569-3

Preston, D., & Carter, M. (2009). A review of the efficacy of the Picture Exchange Communication System intervention. Journal of Autism and Developmental Disorders, 39, 1471–1486. doi:10.1007/s10803-009-0763-y

Remington, B., Hastings, R. P., Kovshoff, H., Espinosa, F. D., Jahr, E., Brown, T., . . .Ward, N. (2007). Early intensive behavioral intervention: Outcomes for children with autism and their parents after two years. American Journal on Mental Retardation, 112, 418–438. doi:10.1352/0895-8017(2007)112[418:EIBIOF]2.0.CO;2

Rogers, S. J., & Vismara, L. A. (2008). Evidence-based comprehensive treatments for early autism. Journal of Clinical Child & Adolescent Psychology, 37, 8–38. doi:10.1080/15374410701817808

Schwartz, I. S., Garfinkle, A. N., & Bauer, J. (1998). The picture exchange communication system: Communicative outcomes for young children with disabilities. Topics in Early Childhood Special Education, 18, 144–159. doi:10.1177/027112149801800305

StataCorp. (2003). Stata statistical software(Version Release 10) [Computer software]. College Station, TX: Author.

Sulzer-Azaroff, B., Hoffman, A. O., Horton, C. B., Bondy, A., & Frost, L. (2009). The Picture Exchange Communication System (PECS): What do the data say?Focus on Autism and Other Developmental Disabilities, 24, 89–103. doi:10.1177/1088357609332743

Webb, T. (2000). Can children with autism and severe learning difficulties be taught to communicate spontaneously and effectively using the Picture Exchange Communication System. Good Autism Practice, 1, 29–41.

Yoder, P., & Compton, D. (2004). Identifying predictors of treatment response. Mental Retardation and Developmental Disabilities Research Reviews, 10, 162–168. doi:10.1002/mrdd.20013

Yoder, P., & Stone, W. L. (2006a). Randomized comparison of two communication interventions for preschoolers with autism spectrum disorders. Journal of Consulting and Clinical Psychology, 74, 426–435. doi:10.1037/0022-006X.74.3.426

Yoder, P., & Stone, W. L. (2006b). A randomized comparison of the effect of two prelinguistic communication interventions on the acquisition of spoken communication in preschoolers with ASD. Journal of Speech, Language, and Hearing Research, 49, 698–711. doi:10.1044/1092-4388(2006/051)

Yoder, P., & Warren, S. F. (1999). Maternal responsivity mediates the relationship between prelinguistic intentional communication and later language. Journal of Early Intervention, 22, 126–136. doi:10.1177/105381519902200205

Submitted: March 25, 2010 Revised: December 28, 2010 Accepted: April 20, 2011

A Measure of Perceived Family Stigma: Validity in a Military Sample

Acknowledgement: This material is based upon work supported in part by a Department of Veterans Affairs Human Services Research and Development Award (PPO 09-314-1 to Julia M. Whealin). The contents do not represent the views of the VA or the United States Government. Thank you to Sandra Bozik-Lyman, MA and Dawna Nelson, MSW for their assistance collecting these data.

Mental health stigma refers to negative attitudes and beliefs about people with mental illnesses, or people who seek mental health services, and is one of the most persistent personal barriers to mental health service utilization. Mental health stigma specifically contributes to prejudice and discrimination against people with mental illness that prevents people from seeking or receiving services (Corrigan & Penn, 1999; Corrigan & Wassel, 2008; Van Brakel, 2006). The present study examined perceived family stigma—an individual’s perceptions or awareness of their family’s negative attitudes and beliefs about the attributes of people with mental illness (see Table 1)—in a predominantly White and Asian American/Pacific Islander (AAPI) sample of U.S. military veterans, as AAPI veterans are at particularly high risk for mental health problems (Tsai & Radhakrishnan, 2012).

Types of Stigma

Various subcategories of mental health stigma exist (see Table 1), the most basic of which are self-stigma and public stigma (Corrigan, 2004). Self-stigma reflects an individual’s personal attitude about stigma, whereas public stigma reflects an individual’s perception about stigma in their external environment (e.g., belief that others will have negative views of an individual with mental illness). Both forms of mental health stigma can have a negative impact on mental health care utilization, given that fear of potential discrimination may discourage treatment seeking (Corrigan & Wassel, 2008), as well as intensify psychosocial difficulties (Livingston & Boyd, 2010). Research measures have been developed to assess various forms of self-stigma and public stigma (Corrigan, Watson, & Barr, 2006; Mak & Cheung, 2010; Ritsher, Otilingam, & Grajales, 2003; Skopp et al., 2012; Struening et al., 2001; Vogel, Wade, & Haake, 2006; Vogel, Wade, & Ascheman, 2009; Vogt, Fox, & Di Leone, 2014). However, current measures of family stigma often conflate it with other sources of perceived public stigma (e.g., Endorsed and Anticipated Stigma Inventory; Vogt et al., 2014), or do not specifically measure perceived family stigma (e.g., Perceived Stigma and Barriers to Care Scale; Britt et al., 2008). As such, it is important that we look at perceived family stigma as its own unique construct both due to how perceived family stigma may be experienced differently than other forms of perceived stigma (Moses, 2010) and the saliency of family in veteran and military cultures.

As family can be an important source of support to those who suffer from mental health issues (Scarpa, Haden, & Hurley, 2006), perceived family stigma may dissuade individuals from seeking help from family and nonfamily members, including professional mental health service providers (Acosta et al., 2014), and act as a significant barrier to mental health service utilization (Vogt et al., 2014). For example, Leaf, Bruce, and Tischler (1986) found that women were less likely to seek treatment if they believed their families would “get upset” at them for doing so. Spoont and colleagues (2014) also found a relationship between stigma from “people in your life” and service use. Although perceived family stigma was not directly measured, Spoont et al. found that the odds of initiating mental health treatment were greater for veterans who were encouraged to seek help by friends and/or family than for those who were not. Thus, the empirical support generally supports a negative relationship between perceived family stigma and likelihood of seeking mental health services.

Over time, individuals may internalize family stigma beliefs as a part of their own value system (Fung, Tsang, & Cheung, 2011). Internalizing requires that individuals adopt others’ (in this case, family members’) attitudes as their own (Corrigan & Rao, 2012). In a study of 300 family members of mentally ill patients (Shrivastava, Johnston, De Sousa, Sonavane, & Shah, 2014), 84.4% of participants reported that observation of family attitudes and/or hearing offensive comments contributed to the patients’ internalizing these negative stereotypes (e.g., internalized stigma, also known as self-stigma). Once internalized, perceived family stigma can lead to negative outcomes including low self-esteem, low empowerment (Livingston & Boyd, 2010) and increased risk for suicide (Griffiths, Crisp, Jorm, & Christensen, 2011). These negative outcomes can be a very real concern given the occurrence of “affiliative stigma” (Chang et al., 2015; Mak & Cheung, 2008), which describes family members’ internalization of public stigma as a result of their affiliation with the individual with mental illness. In other words, affiliative stigma occurs when the family member internalizes their perceived courtesy stigma (Corrigan & Miller, 2004; see Table 1). Alternatively, “reflective stigma” involves situations where individuals believe that their mental illness could cause their family members to experience pubic stigma, although the individual does not necessarily endorse or internalize the stigma themselves. For example, Gilbert, Gilbert, and Sanghera (2004) assessed Asian women via focus groups and found that the primary concern participants had about seeking mental health treatment was the shame that seeking such treatment would bring to loved ones, whereas their personal attitudes (i.e., self-stigma) was much less of a barrier to seeking help.

Perceived family stigma may be particularly salient for some groups over others. AAPIs, for instance, may be more vulnerable to perceived family stigma given the cultural values endorsed by individuals who identify as AAPI. Research has found that, in general, AAPIs endorse values such as maintaining face and have a tendency for a collectivistic orientation, which make it likely for AAPIs to consider their reputation not only a reflection of themselves but also that of their family (Kim, Atkinson, & Umemoto, 2001; Lin & Tsai, 2016). Given the importance of honoring the family in AAPI cultures (Sandhu, 1997), this fear of bringing shame to the family may be a particularly salient barrier to seeking mental health services in this population, even over and above other barriers. Previous research by Yang, Phelan, and Link (2008), which looked at community attitudes of efficacy and shame in Chinese Americans, demonstrated that familial and community stigma may play a role in why they hesitate to seek counseling. More broadly, Golberstein, Eisenberg, and Gollust (2008) found that AAPIs perceived higher levels of stigma and were significantly less likely to use services than Whites. Thus, it is important to identify how perceived family stigma in and of itself (separate from public stigma) differentially impacts utilization in AAPIs. Outside of ethnic/racial differences, there may also be gender differences in perceived family stigma. Research by Ojeda and Bergstresser (2008) suggested that men were more likely than women to endorse items suggesting that fear of stigma from others (e.g., neighbors) was a barrier impacting mental health service seeking decisions. As this finding suggests men may be more likely than women to be concerned about opinions from outside sources, it is likely that this may also apply to the opinions of family members. As such, men may be more likely to report higher perceived family stigma levels, which could potentially impact their mental health service seeking.

In addition to its potential impact on specific sociodemographic groups, perceived family stigma is also expected to be a salient factor on mental health service seeking for Active Duty and veteran populations. A fundamental characteristic across military populations is the focus on self-reliance, as well as the emotional control underlying the warrior way of life, referred to as the “warrior ethos” (Hall, 2008). Values such as stoicism are often prized within Active Duty and veteran cultures, resulting in a decreased likelihood that mental health problems will be discussed or acknowledged. Importantly, the warrior ethos value can extend to family members as well, who are expected to represent their service member positively (Kuehner, 2013). Unlike civilian families, military families are acculturated to military values as a function of many of the characteristics of military life—not having a choice about where you live, often being required to move to new locations every few years, often living in military-only bases, and involvement in social activities (Hall, 2008). When this is the case, family members are likely to adopt the warrior ethos, and, as a result, may covertly and overtly discourage each other from talking about mental health problems (DeCarvalho & Whealin, 2012). This can lead to perceptions that mental health problems would be viewed negatively by family members and fears that mental health problems would reflect poorly on the family, all aspects of perceived family stigma.

Concerns related to perceived family stigma can be highly problematic given the prevalence of mental health sequalae, such as posttraumatic stress disorder (PTSD), that results from combat and other operational stressors (Barrett, Gray, Doebbeling, Clauw, & Reeves, 2002; Hoge et al., 2004; Iversen et al., 2008; O’Toole, Marshall, Schureck, & Dobson, 1999). Of the 289,328 Iraq and Afghanistan veterans who entered the VA health care system between 2002 and 2008, 21.8% were diagnosed with PTSD (Seal et al., 2009). However, despite the high rates of military-related sequelae, most veterans who are diagnosed with a mental health disorder underutilize mental health services. Among U.S. veterans who served in Iraq and Afghanistan, for instance, almost half of those who screened positive for PTSD and/or depression reported that they had not received any treatment in the previous year (Schell & Marshall, 2008). In order to address this gap in mental health care, it is important that we look at the barriers to seeking help. Perceived family stigma is one such important barrier that needs to be studied, both due to its potential salience within the Active Duty and veteran population, as well as previous research that has indicated that stigma-related concerns are one such barrier that influence veterans’ intentions to seek mental health treatment (Acosta et al., 2014; Leaf et al., 1986; Ouimette et al., 2011; Pietrzak, Johnson, Goldstein, Malley, & Southwick, 2009; Vogt et al., 2014). Thus, the influence of the military cultural values (for example, the warrior ethos, which may promote mental health stigma) that permeate the family in combination with the amount of military-related mental health sequelae veterans face, makes perceived family stigma potentially highly relevant to veterans and their mental health service seeking. However, very little of the research has looked at perceived family stigma, so much of our understanding of perceived family stigma is limited. Additionally, previous research on public stigma has often focused on White military populations, with little attention to AAPI military populations, despite the salience of family as a factor impacting treatment seeking within this population. As such, the present study provides much needed information on perceived family stigma in the APPI veteran population.

Given the lack of published measures to assess perceived family stigma, the goal of the present study was (a) develop a short, survey-based measure of perceived family stigma, (b) evaluate the psychometric properties of the Perceived Family Stigma Scale (PFSS), and (c) explore correlates and outcomes (i.e., need or desire for help for emotional problems) related to perceived family stigma. As such, the present study specifically sought to demonstrate the reliability and convergent validity of a self-report scale that exclusively measures perceived family stigma. In addition, this study examined the factor invariance of the items in the PFSS across demographic factors that have been shown to correlate with mental health service use, such as gender, ethnicity/race, marital status, income level, rural versus urban residence, deployment status, and rank in the military (Cohen & Hesselbart, 1993; Fikretoglu, Guay, Pedlar, & Brunet, 2008; Hoge, Auchterlonie, & Milliken, 2006; Lin, Goering, Offord, Campbell, & Boyle, 1996; Mackenzie, Gekoski, & Knox, 2006; Stewart, Jameson, & Curtin, 2015; Sue, Fujino, Hu, Takeuchi, & Zane, 1991). Furthermore, we explored PFSS by examining correlates of PFSS as well as the relationship between PFSS and need or desire for help for emotional problems, as well as whether the relationship was moderated by gender or ethnicity/race. We had no specific hypotheses regarding the differential associations for other demographic groups (i.e., based upon income level, deployment history, education level, rural status, marital status, or rank in the military). As research has found that individuals who feel unable to disclose their emotional problems to significant others have higher levels of mental illness (Belsher, Ruzek, Bongar, & Cordova, 2012; Cordova, Cunningham, Carlson, & Andrykowski, 2001; Lepore, Silver, Wortman, & Wayment, 1996), we hypothesized that veterans with low PFSS scores would be more likely to express a desire for help for an emotional problem compared to those with high PFSS scores, but did not have specific hypotheses whether this relationship would be moderated by gender or ethnicity/race.

Procedure

A survey was conducted among veterans and members of the U.S. Air and Army Guard using a multistage mailing procedure (Mangione, 1998) in 2011. A cover letter and fact sheet explaining the study and consent process were mailed, followed by three reminder letters. An incentive of $20 was included with the survey to provide remuneration for veterans’ time. This research was approved by the VA Pacific Islands Health Care System Institutional Review Board.

Participants

Participants were a Hawaii sample of 623 veterans and national guard members that served in the post September 2001 conflicts formally known as Operation Enduring Freedom (OEF), Operation Iraqi Freedom (OIF), and Operation New Dawn (OND). Because over 40% of veterans serving after September 11, 2001 are from rural areas, samples were stratified by rural/urban residence using VA guidelines (West et al., 2010), in which urban residents are defined as those living in a U.S. Census-defined urbanized area. Rural residents included anyone not defined as urban.

Of a total of 450 veterans randomly selected from Hawaii’s OEF/OIF/OND program registry (with a roster of 2,565 OEF/OIF/OND veterans) who were mailed surveys, 233 (58% response rate) returned completed surveys. Of a total of 800 veterans randomly selected from a list of all enrolled National Guard members in the state of Hawaii (n = 5,300) who were mailed surveys, 390 (49% response rate) returned completed surveys. The total sample from both Hawaii samples was 623 veterans between November 2011 and May, 2012. Following recommended practices (U.S. Department of Health & Human Services, 2011) and methods used in previous studies (Kaneshiro, Geling, Gellert, & Millar, 2011), participants were grouped based on the specific ethnic-racial categories they provided. The sample consisted of 139 East Asians (Chinese/Japanese/Korean), 113 170 Native Hawaiians/Pacific Islanders (NHPI), and 124 Whites. Given the limited number of participants from other ethno-racial groups (e.g., Latinos/Latinas, Blacks), participants who did not fall into one of the four ethnic-racial categories above were all grouped under an Other ethnicity/race category. The two samples did not differ by gender, age, education, rural–urban region, or marital status. Compared to the general population of veteran and National Guard members, the current samples did not differ by age, gender, or marital status, but were more likely to identify as AAPI.

Measures

Perceived Family Stigma Scale (PFSS)

Prior to psychometrically testing the items as a part of the present study, a multistep process was used to develop the PFSS. With this preliminary work, researchers systematically reviewed the literature, created a draft instrument, and refined the scale items.

A scale development team was brought together that consisted of the second author’s research team and diverse mental health scholars (e.g., experts from relevant cultural groups—including military populations, rural populations, and ethnically diverse populations—who had clinical expertise working with those groups). This team reviewed the literature on mental health stigma theory (Corrigan, 2004; Corrigan & Wassel, 2008; Link & Phelan, 2001; Livingston & Boyd, 2010; Van Brakel, 2006), the literature on mental health stigma in diverse groups (Abdullah & Brown, 2011; Atkinson & Gim, 1989; P.W. Corrigan & Miller, 2004; Davis, Ressler, Schwartz, Stephens, & Bradley, 2008; Judd et al., 2006; Yang et al., 2008) including military populations (Hoge et al., 2004; Kim, Thomas, Wilk, Castro, & Hoge, 2010; Pietrzak et al., 2009; Stecker, Fortney, Hamilton, & Ajzen, 2007; Vogt, 2011), and work that assessed stigma experienced by family members of those with mental health problems (Corrigan, 2004; Wahl & Harman, 1989; Struening et al., 2001; Alvidrez, 1999; Murray-Swank et al., 2007; Fung et al., 2011; Shrivastava et al., 2014; Livingston & Boyd, 2010; Stuart, 2008). The team additionally reviewed literature on attitudinal and cultural factors related to seeking help for mental health problems (Alvidrez, 1999; Finley et al., 2010; Gorman, Blow, Ames, & Reed, 2011; Murray-Swank et al., 2007; Ortega & Rosenheck, 2000; Yoshioka, Gilbert, El-Bassel, & Baig-Amin, 2003). The literature review found that although other scales had been developed that assessed stigma experienced by family members of those with mental health problems, no scale had yet been developed that systematically targeted perceived family stigma among individuals with mental health problems.

After verifying that a perceived family stigma construct had not yet been developed, the team collectively identified the concepts that comprised the perceived family stigma construct using a rational approach (Haynes, Richard, & Kubany, 1995). The concepts included perceptions of family secrecy, shame, blame, and doubt in one’s ability because of a mental health disorder. Five items were then developed that reflected the perceived family stigma construct: (a) “People in my family do not talk about mental health problems with people outside of my family”; (b) “It would reflect badly on people in my family if people knew I had a mental health problem”; (c) “People in my family would blame me for the problem”; (d) “People in my family might have less confidence in me”; and (e) “People in my family might treat me differently.” Participants were instructed to rate how much each concern would affect their likelihood of seeking services if they were suffering from a mental/emotional problem. Items and question stems were formatted to be consistent with the wording of existing measures of stigma in military populations (e.g., Britt et al., 2008).

From this work, the second author evaluated the items via telephone interviews with 40 veterans who had been diagnosed with PTSD as a part of a larger project (Whealin et al., 2012). Veterans were read the items and asked to describe what each item was assessing, thus judging whether items were understandable, culturally appropriate, and reflective of the intended construct. The assessor then determined whether the veteran easily comprehended the construct intended by each item (yes/no). Based on interview feedback, Item (e), “People in my family might treat me differently,” was determined to be unclear and redundant to the other items and was removed from the scale. Additionally, Item (b), “It would reflect badly on people in my family if people knew I had a mental health problem,” was clarified to read, “My family would look bad or be ashamed if people knew I had a mental health problem.”

The aim of the focus groups was to identify individual, family, and community factors that facilitate or deter veterans’ mental health service seeking rather than to exclusively assess perceived family stigma. After piloting the procedure at three sites, veterans and family members were recruited from five targeted locations in the rural Pacific Islands. Purposeful sampling was used to ensure that veterans and family members who were representative of different war eras were included at each location. Six veteran (N = 37) and six family member (N = 38) focus groups were conducted using a semistructured interview guide (Whealin, Nelson, Mahoney, & Kawasaki, 2016). Focus groups were transcribed verbatim using a Grounded Theory approach to guide constructs and verify resultant themes (Glaser & Strauss, 1967). Perceived family stigma emerged as one construct within a “Family Facilitators and Barriers” theme and consisted of the need to keep mental health problems within the family and the perception that family members thought less of them because of their mental health problem(s).

Drawing from the telephone interviews and focus groups, the second author then met with cultural experts who further refined questions so that items were consistent with the terminology of diverse veteran populations. For example, interview and focus group members often described PTSD simply as stress and used the term problem to refer to a disorder. As a result, we purposefully replaced disorder and PTSD with terms that reflected participants’ usage and comprehension.

Perceived Stigma and Barriers to Care Scale (PSBCS)

The PSBCS (Britt et al., 2008) consists of 13 items that query respondents to: “Rate each of the possible concerns that might affect your decision to receive mental health counseling or services if you ever had a problem” (Hoge et al., 2004). Items measure obstacles that prevent or dissuade individuals from seeking mental health treatment (e.g., “I don’t have good transportation”), as well as self-stigma (“It would be too embarrassing”) and public stigma (“I would be seen as weak”). Factor analyses in veteran and active duty samples have revealed a two-factor solution: self and public stigma-based barriers to care (8 items) and logistical barriers to care (5 items). Responses are measured on a 5-point Likert-type scale (1 = Strongly Disagree to 5 = Strongly Agree). We calculated the mean response to items for a total score and for the two subscales, with higher scores reflecting greater perceived barriers to care. Comparison data were available for N = 6153 combat veterans, including stratified samples of soldiers who met screening criteria for psychopathology (n = 731) and those who did not (n = 5422) following deployments to Iraq and Afghanistan (Hoge et al., 2004). The internal consistency for this scale was excellent (α = .92).

Devaluation of Consumers Scale (DCS)

The DCS (Struening et al., 2001) is an eight-item scale based on Link, Cullen, Struening, Shrout, and Dohrenwend’s (1989) original scale that examined perceptions of community attitudes toward people with mental illness (i.e., public stigma). Participants rate their level of agreement with eight statements, such as “Most people would accept a person who once had a serious mental illness as a close friend.” Responses are measured on a 4-point Likert-type scale (1 = Strongly Disagree to 4 = Strongly Agree). We summed responses to all items for a total score ranging from 8 to 32, with higher scores reflecting greater perceptions of community stigma. Internal consistency of DCS score was adequate in this sample (α = .66) but slightly worse than in a previous sample (α = .82; Struening et al., 2001).

Need or want for help for emotional problems

One item was used from The Self-Help and Treatment Services Utilization Survey (Fontana et al., 2006)—a psychometrically sound 20-item scale that assesses perceived need for health care services—to assess for need or want for help for emotional problems: “In the last 3 months, have you needed or wanted help for an emotional problem (such as stress, anger, or depression).” Participants answered either “yes” or “no”.

Data Analysis

First, the concurrent and incremental validity of the PFSS was tested. To test the concurrent validity of the PFSS, the PFSS was correlated with the PSBCS and the DCS, both of which are established measures of stigma and barriers to care. To demonstrate its use as a measure of a barrier to help seeking, the incremental validity of the PFSS over the PSBCS was tested using hierarchical logistic regression. Using hierarchical logistic regression, two models were compared, one with just the PSBCS regressed on the binary item from the Self-Help and Treatment Services Utilization Survey assessing perceived need for mental health care services (“In the last 3 months, have you needed or wanted help for an emotional problem [such as stress, anger, or depression]?”), and the second model that also included the PFSS. The PFSS would be determined to have incremental validity if PFSS significantly predicted likelihood of needing or wanting help for emotional problems over and above the PSBCS.

An analysis of variance (ANOVA) was also used to explore the correlates of the PFSS using SPSS, where the PFSS was set as the dependent variable and eight categorical independent variables—gender, ethnicity/race, income level, deployment history, education level, urban/rural location, marital status, and military rank—were identified. Due the limited number of research conducted on perceived family stigma thus far, a more exploratory approach was implemented in order to better understand how perceived family stigma might differ across various demographic groups. As such, in addition to the main effects, the model tested included all two-way interactions between the independent variables; more complex interactions (i.e., three-way) were not included in the model. Effect sizes were reported using ηp² using cut-off scores following those established by Cohen (1988) as well as Olejnik and Algina (2000), where values of .01 indicate a small effect size, .06 a medium effect size, and .14 a large effect size. The relationship between the PFSS and the likelihood of participants needing or wanting help for an emotional problem was also tested using SPSS. A logistic regression was performed with desire for help for emotional problems as the dependent variable, and PFSS as the independent variable. To test whether gender or race moderated the relationship between PFSS and need or want for help, the model also included a two-way interaction between PFSS and gender as well as a two way-interaction between PFSS and ethnicity/race. All missing data were handled using the SPSS default of list wise deletion.

Multigroup categorical confirmatory factor analysis (CFA) was used to test the measurement invariance of four items that comprised the PFSS across gender, ethnicity/race, income level, deployment history, education level, urban/rural location, marital status, and military rank, following the guidelines set by Muthén and Asparouhov (2002). A single factor four-item model was specified for all steps of the measurement invariance test utilizing weighted least squares means and variance adjusted (WLSMV) estimation with THETA parameterization in Mplus (Muthén & Muthén, 2012). WLSMV estimation was utilized as previous research has suggested that treating ordered categorical data as continuous data would lead to distorted results, particularly when doing measurement invariance analyses (Lubke & Muthén, 2004). Thus, to make meaningful group comparisons, categorical data should be analyzed with models meant for categorical outcomes. Additionally, in order to avoid response categories with low endorsement rates (e.g., less than 5% response rate), categories with low response rates were collapsed with either the preceding or following category. As the strongly disagree and strongly agree categories of the indicator variables had a less than 5% response rate, the strongly disagree and disagree categories were combined to form one category, and the strongly agree and agree categories were combined as well to form their own category. Thus, the PFSS items were rescaled as 5-point response items.

All missing data were handled using pairwise deletion, as the WLSMV estimation does not allow the use of full information maximum likelihood (FIML) to handle missing data. For the configural invariance model, factor loadings and thresholds were both freely estimated, while the factor mean and variance were constrained to 0 and 1 in all groups for identification. For the measurement invariance model, factor loadings were constrained to equivalency and thresholds remained free; the factor mean was constrained to 0 in the target group, but freely estimated in the alternative group(s) while the factor variance was constrained to 1 in all groups for identification. For the scalar invariance model, both factor loadings and thresholds were constrained to equivalency, while the factor mean and variance were constrained to 0 and 1, respectively in the target group, but freely estimated in the alternative group(s) for identification purposes.

The overall fit at each step was determined by the model’s statistical and descriptive fit. A model is determined to have acceptable fit if it meets the cut-off scores indicating acceptable model fit for the descriptive fit indices: the comparative fit index (CFI), Tucker-Lewis Index (TLI), and the root mean square error of approximation (RMSEA). Utilizing the cut-off scores proposed by Bentler (1990); Hu and Bentler (1999), and Steiger (1990), (a) CFI values greater than .95 indicated good model fit and values greater than .90 indicated acceptable model fit; (b) TLI values greater than .95 indicated good model fit; and (c) RMSEA values less than .05 indicated good model fit and values less than .08 indicated acceptable model fit. For configural invariance only, just acceptable model fit was necessary to establish configural invariance. If configural invariance could not be established, the fit of the model across the baseline groups were examined separately instead. However, for metric and scalar invariance to be established there must be acceptable model fit and each model also has to demonstrate that it fits the data as well as the preceding model. This was done with a chi-square difference test using nested model comparisons through the DIFFTEST procedure in Mplus, as it is not possible to do a traditional chi-square difference test when using WLSMV estimation. The idea is to find a more parsimonious model (i.e., more restrained invariance model) without significant fit loss. If invariance could not be established, the modification indices will be examined, and revised invariance models will be tested to see if partial invariance could be established following procedures set by Steenkamp and Baumgartner (1998).

In the case that noninvariance is found, the effect size of the noninvariance on the perceived family stigma scores will be examined using the latent trait estimation approach suggested in Tay, Meade, and Cao (2015). It is a simple heuristic approach to determine if the presence of invariance affects the interpretation of whether reference and focal groups differ in latent trait levels substantially. To do so, the latent mean differences between the configural and scalar models will be calculated, with the differences interpreted similar to Cohen’s d (.2 = small, .5 = medium, .8 = large). As latent means are constrained in the model specifications used in the tests of measurement invariance, a different set of parameters will be used to obtain the latent means. DELTA parameterization and WLSMV estimation will be used for all analyses. For the configural model, factor means and variances will be constrained to 0 and 1, respectively, in the target group but freed in the alternative group(s). Additionally, the factor loading of the reference variable will be constrained to 1 across both the target and alternative group(s), the factor loadings of other variables are kept free. All thresholds of the reference variable are held to equivalency, while only the first thresholds of all other items will be constrained to equivalency. For the scalar model, the factor loading of the reference variable will be constrained to one, while the factor loadings of other variables will be constrained to equivalency across groups. All thresholds will be held to equivalency, and the factor means and variances will be constrained to 0 and 1, respectively, in the target group but freed in the alternative group(s).

The PFSS had acceptable internal consistency (α = .86) as well as strong interitem correlations (values ranged from r = .51 to .76; see Table 2). The PFSS also correlated with the Perceived Stigma and Barriers to Care Scale (PSBCS) (r[588] = .68, p < .001) and the Devaluation of Consumers Scale (DCS) (r[588] = .44, p < .001) to demonstrate concurrent validity. Moreover, PFSS significantly predicted likelihood of needing or wanting help for emotional problems (p < .001) over and above the PSBCS (p = .562), which demonstrated the incremental validity of the PFSS over the PSBCS. The CFA results indicated that the single factor model was a good fit statistically, χ²(2, N = 620) = .34, p = .84 and descriptively (CFI = 1.00, TLI = 1.00, RMSEA < .001) to the PFSS, indicating a good overall fit to the data irrespective of group membership. The standardized factors loadings were large and statistically significant (.759 to .971).

Family Stigma Scale Interitem Correlation

Measurement Invariance

The measurement invariance results indicated that the PFSS had full measurement invariance across deployment history, education level, urban/rural location, marital status, and military rank. However, the PFSS demonstrated only partial measurement invariance across gender, ethnicity/race, and income level. Across gender, configural invariance was achieved, but full metric invariance was not. An examination of the modification indices revealed that the factor loading for the “people in my family do not talk about mental health problems with people outside our family” (fam_talk) item contributed the most to the worse fit of the metric invariance model in comparison to the configural invariance model. Across ethnicity/race, the PFSS demonstrated configural and metric invariance, but not scalar invariance. An examination of the modification indices revealed that the second threshold for the “people in my family would blame me for the problem” (fam_blame) item for the White ethnic group was the most problematic. Across income, the PFSS demonstrated configural invariance but not metric invariance. An examination of the modification indices revealed that the factor loading for the fam_talk item for participants who made more than $50,000 was the most problematic. See Table 3 for the measurement invariance results and Table 4 for standardized factor loadings.

Invariance Model Results by Group

Standardized Item Factor Loadings (Standard Errors) for All Groups

Finally, the effect size of the noninvariance across gender, ethnicity, and income was calculated. For gender, the female group was set as the target group. The latent mean for males in the configural model was .20 and was .27 in the scalar model (Δ = .07). Across ethnicity, the East Asian ethnic group was set as the target group. The latent means in the configural model were −.07, −.23, −.30, and were −.10, −.19, −.49 in the scalar model for the Filipino (Δ = −.03), NHPI (Δ = .04), and White (Δ = −.19) groups respectively. Across the income groups, the less than $35,000 income group was set at the target group. The latent means for the income group between $35,000 and $50,000 was −.08 in the configural model and −.11 in the scalar model, (Δ = −.03). For the greater than $50,000 income group, the latent mean was −.11 in the configural model and .01 in the scalar group (Δ = .12).

Demographic Correlates

There was a significant interaction between urban/rural location and deployment history, F(1, 419) = 5.97, p = .015, ηp² = .014. For participants who had never been deployed, the estimated marginal mean PFSS scores were higher among those who currently live in a rural area compared to an urban area (M = 1.88 vs. M = 1.69). For participants who had previously been deployed to a combat area, the estimated marginal mean PFSS scores were lower for those who live in rural areas compared to those who live in urban areas (M = 1.73 vs. M = 1.90). Similarly, there was a significant interaction between income level and education level, F(4, 419) = 4.06, p = .003, ηp² = .037. For participants who were high school graduates or did not complete high school, estimated marginal mean PFSS scores were higher when participants earned between $35,001 and $50,000 than when participants earned less than $35,000 (M = 1.67 vs. M = 1.91), although estimated marginal mean PFSS scores were lower when participants earned more than $50,000 (M = 1.84). For participants who had at least some college-level education or were a college graduate, estimated marginal mean PFSS scores increased as income increased (less than $35,000 [M = 1.50], $35,001–$50,000 [M = 1.66], more than $50,000 [M = 1.78]). Participants who had more than a college graduate education level (i.e., graduate school) had the highest estimated marginal mean PFSS scores when they made less than $35,000 (M = 2.59) compared to those who made between $35,001 and $50,000 (M = 1.25) and those who made more than $50,000 (M = 1.99). There were no other significant interactions.

In addition to the significant interaction, there was one statistically significant main effect by rank, F(2, 419) = 3.87, p = .022, ηp² = .018. An examination of the estimated marginal means revealed that the mean PFSS scores of those who were enlisted were significantly higher than the mean PFSS scores of those who were senior enlisted, all else being equal, M = 1.87 versus M = 1.44, p = .026. See Table 5 for the estimated marginal means of the main effects. Furthermore, while PFSS scores of different ethnic/racial groups were not found to differ significantly, pairwise comparisons indicated that those who identified as Filipino did report significantly higher PFSS levels than Asians (Korean, Chinese, or Japanese; p = .043) and Native Hawaiian/Pacific Islanders (p = .029).

Estimated Marginal Means for Main Effects

Need for Help for Emotional Problems

There was a significant interaction between gender and PFSS, controlling for ethnicity/race, p = .018. This suggests that while an increase in mean PFSS scores predicted a higher probability of report of a need or want for help for an emotional problem for both men and women, for women each 1-point increase in mean PFSS scores predicted a likelihood of reporting a need/desire for help for an emotional problem 3.077 times that of men (see Figure 1). There was no significant interaction between ethnicity/race and mean PFSS scores. Furthermore, results indicated significant main effects by PFSS and gender, although these findings are tempered by the aforementioned significant interaction. Specifically, the odds that the participant would report a need or want for help for an emotional problem increased by 3.80 times, all else being equal, for every 1-point increase in PFSS, p < .001. The odds of men reporting a need or want for help for an emotional problem also were 6.34 times that of women’s, p = .020. There was no significant main effect of ethnicity/race.

Figure 1. Predicted log odds of reporting a need or want for help for an emotional problem.

Until now, there has been a lack of a valid and reliable measure of perceived family stigma, an individual’s perceptions of familial attitudes and beliefs about mental illness and/or people with mental illness. Instead, previous measures of perceived family stigma have often been conflated with measures of other sources of public stigma. Given that stigma from families may be experienced differently than stigma from other sources (Moses, 2010), a measure of perceived family stigma is a needed. The present study provides preliminary support for the reliability and psychometric validity of the PFSS. The PFSS demonstrated concurrent validity with a multidimensional measure of mental health care barriers widely used for military populations (the PSBCS; Britt et al., 2008) and a measure of public stigma (the DCS; Struening et al., 2001). The PFSS was also significantly correlated with reported need/desire for help for an emotional problem, providing additional evidence that the scale measures the construct it was intended to measure. Importantly, the PFSS demonstrated incremental validity over the PSBCS, suggesting that the PFSS may be a useful measure, particularly when examining the relationship between perceived family stigma and service use in veteran populations.

As hypothesized, higher scores on the PFSS were associated with greater likelihood of wanting/needing help for an emotional problem. Men were 6 times more likely than women to report a need for help with an emotional problem. However, as PFSS scores increased, this disparity decreased until women responded as more likely than men to report a need for help (see Figure 1). This finding is in line with previous research that has found that individuals who feel unable to disclose their emotional problems to significant others (in this case family members) are more likely to report having mental health concerns (Belsher et al., 2012; Cordova et al., 2001; Lepore et al., 1996). Previous research by Leaf and colleagues (1986), where 3,058 community members were interviewed found that concerns about family members’ reactions to their seeking mental health treatment help dissuaded women, with mental health problems from initiating treatment. This suggests that, despite indicating greater need or desire for help with emotional problem as suggested by the present study, individuals who experience high perceived family stigma may not be seeking the help that they need. Given that the present study did not actually look at subsequent treatment seeking, research that evaluates the role of perceived family stigma in predicting an unmet need for mental health treatment may prove to be a useful next step in identifying barriers to care.

The demographic variations in perceived family stigma are also of interest. Perceived family stigma levels were not found to differ significantly across broad ethnic/racial groups, which is consistent with previous work that examined veterans (e.g., Pietrzak et al., 2009). However, pairwise comparisons did indicate that veterans that identified as Filipino reported significantly higher perceived family stigma than those who identified as East Asian (Chinese, Japanese, Korean) or Native Hawaiian/other Pacific Islander. The present study highlights the importance of examining nonaggregated APPI populations. Although little research has been done looking specifically at mental health utilization and barriers in Filipino veterans, Sanchez and Gaw (2007) argued that, in traditional Filipino culture, an individual’s mental illness is reflected upon the entire family and is “associated with shame and stigma.” This suggests that perceived family stigma may be a particularly influential factor for Filipinos when making decisions about mental health care compared to other AAPI veterans groups, and warrants further investigation.

Perceived family stigma levels were also shown to differ significantly across military rankings. Senior enlisted veterans had the lowest perceived family stigma levels, followed by enlisted veterans, with officers reporting the highest perceived family stigma levels among the three groups. Although enlisted veterans had significantly higher PFSS scores than senior enlisted veterans, perceived family stigma was not significantly different between senior enlisted and officers or between enlisted and officers. Previous research has demonstrated that higher rank is associated with lower interest in receiving help for a mental health problem in veterans with mental health disorders (Brown, Creel, Engel, Herrell, & Hoge, 2011), suggesting that higher ranked individuals may perceive greater stigma and would be less willing seek help for their issues. However, most previous research has not separated enlisted from senior enlisted staff when examining stigma, the latter being a distinct subgroup with higher military rank (e.g., E5–E9 vs. E1–E4) than other enlisted service members and, often, greater longevity compared to both regular enlisted and officers. In addition to military rank, there were significant differences in perceived family stigma when taking into consideration the interactions between rural/urban residency and deployment history and between income and education levels. Whereas more research is needed to understand why there were significant differences in PFSS scores across these groups, these results indicate that some groups may be at risk of presenting with higher marginal PFSS mean scores, which could impact their likelihood of initiating mental health treatment (Spoont et al., 2014). It may be of interest for future researchers to also look at the impact of other factors that can influence education level, such as disability.

In the present study, there were no significant differences in mean PFSS scores across gender and socioeconomic status. The nonsignificant findings regarding gender are consistent with other research that has examined military populations (Elnitsky et al., 2013), but contrasts with Ojeda and Bergstresser’s (2008) study that found civilian men were more likely than civilian women to report perceived public stigma as a barrier to mental health service seeking. It is possible that the nonsignificant finding regarding gender in military populations could be due to the unique characteristics of the population. This explanation is supported by Hourani, Williams, Bray, Wilk, and Hoge (2016), who found that male and female active duty personnel have comparable rates of service utilization, which the authors suggested could be due to similar levels of stigma and access to care between men and women. Although research has not examined between-group SES differences in military samples, the present findings are consistent with research that has found no correlation between income and perceived stigma in nonmilitary populations (Ojeda & Bergstresser, 2008).

As the PFSS was a new scale, one of the purposes of the present study was also to test its invariance across gender, ethnicity/race, marital status, income level, rural versus urban residence, deployment status, and rank in the military. The scale was invariant across marital status, rural versus urban residence, deployment status, and rank in the military, which suggests that the latent variable of perceived family stigma is conceptualized similarly across those who are of different marital status, those who live in rural or urban residences, those who have previously been deployed and those who have not, as well as across those who are of different military ranks. As such, latent factor means and variances, as well as standardized measures of association can be compared across these groups. Additionally, findings suggests that any differences in latent means across these groups is a reflection of true latent mean differences rather than differences in observed item means.

The PFSS met partial but not full invariance across gender, ethnicity/race, and income. Thus, comparisons of latent factor variances or means across these groups need to take into consideration this lack of invariance. For example, the lack of full metric invariance indicates that the units of measurement are different across groups, and that difference score comparisons may not be valid as the items were using nonequivalent units of measurement. Similarly the lack of full scalar invariance suggests that differences in latent variable means may not truly reflect differences in the latent variable but the observed variable instead. However, given the small effect sizes of the lack of invariance, it is likely that valid comparisons of perceived family stigma latent means and variances could be made across gender, ethnicity/race, and income. Additionally, as previous literature has indicated that meaningful comparisons can be made if there is partial invariance present (Byrne, Shavelson, & Muthén, 1989), the authors would argue that PFSS scores could be compared across the subgroups tested in the present study, while taking the following into consideration.

Partial metric invariance was achieved across both gender and income. Notably, there was only one item—“people in my family do not talk about mental health problems with people outside our family”—that was noninvariant across both gender and income groups. All other items demonstrated full factor invariance. Across gender, this item loaded more strongly onto the latent variable of perceived family stigma for men than for women, while across income the item loaded more strongly onto the latent variable for those who made more than $50,000. This set of findings suggest that the “people in my family do not talk about mental health problems with people outside our family” item was more representative of the perceived family stigma latent variable for men than for women, and for those who make more than $50,000 when compared to those who make less than $35,000 or between $35,000 and $50,000. As only partial metric invariance could be established, only partial scalar invariance could be established across gender and income as well.

One possibility for the lack of invariance is due to gender differences in willingness to disclose mental issues. For example, boys are less likely to talk with friends and nonparent family members about their emotional problems than girls and are more likely to report higher levels of perceived stigma (Chandra & Minkovitz, 2006), as well as having less willingness to disclose mental issues or emotional problems (Jorm & Wright, 2008; Su, Wang, & Lin, 2013). Given their greater unwillingness to disclose mental illnesses—particularly to nonfamily members—men may be more likely to anticipate stigma as a result of disclosure than are women. Taking into consideration the limited number of studies that have explored the relationship between income and perceived family stigma, it is difficult to conclude whether there are differences between different income brackets in willingness to disclose mental issues with nonfamily members. One possible explanation for this finding may come from parenting research, which showed that parents of low SES families were less likely to talk with their children than parents of high SES families (Gottfried & Gottfried, 1984). Since there is more communication in high SES families, the lack of communication may then be interpreted as disapproval or stigma against mental illnesses.

While full metric invariance was established across the ethno-racial groups, full scalar invariance could not be established. When latent means were equivalent, White participants had a slightly higher probability of agreeing that perceived blame from their family would affect whether they attended mental health counseling than participants of other ethnic groups. This finding is of particular interest as, considering the importance Asian American and NHPI groups place on family (Julian, McKenry, & McKelvey, 1994), we would expect that Whites would either show an equivalent or lower probability of agreeing that perceived blame from their family would affect whether they attended mental health counseling. However, to the best of the authors’ knowledge, there has yet to be any research that has examined how the relationship between ethnicity/race and perceived blame from family members impacts mental health utilization. As such, it is uncertain what factors are contributing to the lack of scalar noninvariance, although one potential reason could be due to underlying ethnic group differences in mental health attitudes between Whites and Asian American/NHPI groups (Fogel & Ford, 2005; Lam, Tsang, Chan, & Corrigan, 2006; Leong & Lau, 2001).

There were a number of limitations that should be noted about the current study. First, the study sampled veterans residing in Hawaii and, as a result, may not generalize to veterans residing in other geographic regions or to nonveteran populations. Military culture can have a strong impact on military families and how mental health problems are handled (Hall, 2008; DeCarvalho & Whealin, 2012). It is important to keep in mind that in military and other cultural subgroups (Yang et al., 2008), the reputation of individuals is seen as a reflection of their families. Thus, it is probable that perceived family stigma may play a more salient or unique role in veteran populations than in nonveteran populations, where community and peers may play a more or as salient role as family. Furthermore, measurement invariance was tested only across White, Native Hawaiian/Pacific Islander, Filipino, and East Asian subgroups due to the limited number of participants of other ethnicities/races. Moreover, while full invariance is not required to make meaningful interpretations, the lack of full invariance could have influenced the results, although the influence is likely to be minimal given the small effect sizes of the noninvariance. Also, while the present study has shown that there are differences in levels of perceived family stigma across different demographic groups (i.e., rank), the mechanisms contributing to these differences are still unknown. The relationship between perceived family stigma, reported mental health service need, and mental health service utilization is a key point that future research needs to address.

Future research should continue development of the PFSS as well as testing its psychometric properties. Although the purpose of the present study was to develop a short survey given the ease of implementation of brief scales in both clinical and research settings, the limited number of items generated is also a limitation of the present study. While perceived family stigma was theorized as consisting of both public and self-stigma and the PFSS was comprised of items that addressed both dimensions, the limited number of items only allowed for the analysis of PFSS as an unidimensional construct. As such, it is probable that the scale would benefit from further expansion, such as the addition of more items that would address unique aspects of family stigma, which would allow for more complex exploration of perceived family stigma as a multidimensional construct. Similarly, though the present study collected information on logistic barriers to care, self-stigma, and public stigma through the PSBCS, we were only able to test the incremental validity of the PFSS over the PSBCS as we did not collect data on a specific form of self or public stigma, such as peer stigma. Thus, one limitation of the present study was the lack of exploration of the discriminate validity of the PFSS from these other specific forms of self and public stigma. However, as the main purpose of the present study was to develop an initial scale of family stigma, the study was mainly exploratory in nature. As such, the scale developed was not meant to be a finalized scale, but rather serve as the foundation upon which further research on family stigma can be built. Despite these limitations, the PFSS proved to be a valid instrument that could be used to further aid research on perceived family stigma.

In conclusion, future research should (a) examine whether the present findings can be replicated in civilian and other veteran populations, (b) continue to test the factor invariance of the PFSS, preferably across demographic groups that the present study was not able to test, (c) test whether perceived family stigma may predict service seeking, and (d) continue to develop the PFSS and test its psychometric properties. Given the importance of understanding the impact of family stigma on the need for help as well as subsequent seeking of such help, measurements and instruments for understanding stigma are needed. The present study helps to validate the PFSS for use with military veterans. Moreover, the present study is the first to examine how perceived family stigma correlates with veterans’ reported need/desire for help with an emotional problem. Despite its limitations, the findings of the study highlight the importance of more fully understanding family stigma. Given the dearth of research on the impact of family stigma, follow-up research is needed to replicate current findings and to evaluate whether family stigma influences disclosure of mental health problems, internalized stigma, and mental health treatment initiation. With this study, it is our hope that future research will look further at perceived family stigma and its potential impact on decreasing the gap between those who need help and those who successfully obtain help for mental health disorders.

Abdullah, T., & Brown, T. L. (2011). Mental illness stigma and ethnocultural beliefs, values, and norms: An integrative review. Clinical Psychology Review, 31, 934–948. 10.1016/j.cpr.2011.05.003

Acosta, J., Becker, A., Cerully, J. L., Fisher, M. P., Martin, L. T., Varadvas, R., . . . Schell, T. L. (2014). Mental health stigma in the military. Santa Monica, CA: RAND Corporation.

Alvidrez, J. (1999). Ethnic variations in mental health attitudes and service use among low-income African American, Latina, and European American young women. Community Mental Health Journal, 35, 515–530. 10.1023/A:1018759201290

Atkinson, D. R., & Gim, R. H. (1989). Asian-American cultural identity and attitudes toward mental health services. Journal of Counseling Psychology, 36, 209–212. 10.1037/0022-0167.36.2.209

Barrett, D. H., Gray, G. C., Doebbeling, B. N., Clauw, D. J., & Reeves, W. C. (2002). Prevalence of symptoms and symptom-based conditions among Gulf War veterans: Current status of research findings. Epidemiologic Reviews, 24, 218–227. 10.1093/epirev/mxf003

Belsher, B. E., Ruzek, J. I., Bongar, B., & Cordova, M. J. (2012). Social constraints, posttraumatic cognitions, and posttraumatic stress disorder in treatment-seeking trauma survivors: Evidence for a social-cognitive processing model. Psychological Trauma: Theory, Research, Practice, and Policy, 4, 386–391. 10.1037/a0024362

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. 10.1037/0033-2909.107.2.238

Britt, T. W., Greene-Shortridge, T. M., Brink, S., Nguyen, Q. B., Rath, J., Cox, A. L., . . .Castro, C. A. (2008). Perceived stigma and barriers to care for psychological treatment: Implications for reactions to stressors in different contexts. Journal of Social and Clinical Psychology, 27, 317–335. 10.1521/jscp.2008.27.4.317

Brown, M. C., Creel, A. H., Engel, C. C., Herrell, R. K., & Hoge, C. W. (2011). Factors associated with interest in receiving help for mental health problems in combat veterans returning from deployment to Iraq. Journal of Nervous and Mental Disease, 199, 797–801. 10.1097/NMD.0b013e31822fc9bf

Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456. 10.1037/0033-2909.105.3.456

Chandra, A., & Minkovitz, C. S. (2006). Stigma starts early: Gender differences in teen willingness to use mental health services. Journal of Adolescent Health, 38(6), 754.e1–754.e8. 10.1016/j.jadohealth.2005.08.011

Chang, C. C., Su, J. A., Tsai, C. S., Yen, C. F., Liu, J. H., & Lin, C. Y. (2015). Rasch analysis suggested three unidimensional domains for Affiliate Stigma Scale: Additional psychometric evaluation. Journal of Clinical Epidemiology, 68, 674–683. 10.1016/j.jclinepi.2015.01.018

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Cohen, P., & Hesselbart, C. S. (1993). Demographic factors in the use of children’s mental health services. American Journal of Public Health, 83, 49–52. 10.2105/AJPH.83.1.49

Cordova, M. J., Cunningham, L. L., Carlson, C. R., & Andrykowski, M. A. (2001). Social constraints, cognitive processing, and adjustment to breast cancer. Journal of Consulting and Clinical Psychology, 69, 706–711. 10.1037/0022-006X.69.4.706

Corrigan, P. (2004). How stigma interferes with mental health care. American Psychologist, 59, 614–625. 10.1037/0003-066X.59.7.614

Corrigan, P. W., & Miller, F. E. (2004). Shame, blame, and contamination: A review of the impact of mental illness stigma on family members. Journal of Mental Health, 13, 537–548. 10.1080/09638230400017004

Corrigan, P. W., & Penn, D. L. (1999). Lessons from social psychology on discrediting psychiatric stigma. American Psychologist, 54, 765–776. 10.1037/0003-066X.54.9.765

Corrigan, P. W., & Rao, D. (2012). On the self-stigma of mental illness: Stages, disclosure, and strategies for change. The Canadian Journal of Psychiatry, 57, 464–469.

Corrigan, P. W., & Wassel, A. (2008). Understanding and influencing the stigma of mental illness. Journal of Psychosocial Nursing and Mental Health Services, 46, 42–48. 10.3928/02793695-20080101-04

Corrigan, P. W., Watson, A. C., & Barr, L. (2006). The self-stigma of mental illness: Implications for self-esteem and self-efficacy. Journal of Social and Clinical Psychology, 25, 875–884. 10.1521/jscp.2006.25.8.875

Davis, R. G., Ressler, K. J., Schwartz, A. C., Stephens, K. J., & Bradley, R. G. (2008). Treatment barriers for low-income, urban African Americans with undiagnosed posttraumatic stress disorder. Journal of Traumatic Stress, 21, 218–222. 10.1002/jts.20313

DeCarvalho, L. T., & Whealin, J. M. (2012). Healing stress in military families. Hoboken, NJ: Wiley. 10.1002/9781118232095

Elnitsky, C. A., Chapman, P. L., Thurman, R. M., Pitts, B. L., Figley, C., & Unwin, B. (2013). Gender differences in combat medic mental health services utilization, barriers, and stigma. Military Medicine, 178, 775–784. 10.7205/MILMED-D-13-00012

Fikretoglu, D., Guay, S., Pedlar, D., & Brunet, A. (2008). Twelve month use of mental health services in a nationally representative, active military sample. Medical care, 46, 217–223. 10.1097/MLR.0b013e31815b979a

Finley, E. P., Zeber, J. E., Pugh, M. J. V., Cantu, G., Copeland, L. A., Parchman, M. L., & Noel, P. H. (2010). Postdeployment health care for returning OEF/OIF military personnel and their social networks: A qualitative approach. Military Medicine, 175, 953–957. 10.7205/MILMED-D-10-00040

Fogel, J., & Ford, D. E. (2005). Stigma beliefs of Asian Americans with depression in an internet sample. The Canadian Journal of Psychiatry, 50, 470–478.

Fontana, A., Ruzek, J., McFall, M., Rosenheck, R., Allen, S., Maxwell, M., & Wahlberg, L. (2006). The Veterans Affairs Military Stress Treatment Assessment (VAMSTA): A descriptive and monitoring instrument for the treatment of PTSD and comorbid disorders. Washington, DC: U.S. Government Printing Office.

Fung, K. M. T., Tsang, H. W. H., & Cheung, W. M. (2011). Randomized controlled trial of the self-stigma reduction program among individuals with schizophrenia. Psychiatry Research, 189, 208–214. 10.1016/j.psychres.2011.02.013

Gilbert, P., Gilbert, J., & Sanghera, J. (2004). A focus group exploration of the impact of izzat, shame, subordination and entrapment on mental health and service use in South Asian women living in Derby. Mental Health, Religion & Culture, 7, 109–130. 10.1080/13674670310001602418

Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine Publishing Company.

Golberstein, E., Eisenberg, D., & Gollust, S. E. (2008). Perceived stigma and mental health care seeking. Psychiatric Services, 59, 392–399. 10.1176/appi.ps.59.4.392

Gorman, L. A., Blow, A. J., Ames, B. D., & Reed, P. L. (2011). National Guard families after combat: Mental health, use of mental health services, and perceived treatment barriers. Psychiatric Services, 62, 28–34. 10.1176/ps.62.1.pss6201_0028

Gottfried, A. W., & Gottfried, A. E. (1984). Home environment and cognitive development in young children of middle-socioeconomic-status families. In A. W.Gottfried (Ed.), Home environment and early cognitive development: Longitudinal research (pp. 57–115). New York, NY: Academic press.

Griffiths, K. M., Crisp, D. A., Jorm, A. F., & Christensen, H. (2011). Does stigma predict a belief in dealing with depression alone?Journal of Affective Disorders, 132, 413–417. 10.1016/j.jad.2011.03.012

Hall, L. K. (2008). Counseling military families: What mental health professionals need to know. New York, NY: Routledge.

Haynes, S. N., Richard, D., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7, 238–247. 10.1037/1040-3590.7.3.238

Hoge, C. W., Auchterlonie, J. L., & Milliken, C. S. (2006). Mental health problems, use of mental health services, and attrition from military service after returning from deployment to Iraq or Afghanistan. JAMA, 295, 1023–1032. 10.1001/jama.295.9.1023

Hoge, C. W., Castro, C. A., Messer, S. C., McGurk, D., Cotting, D. I., & Koffman, R. L. (2004). Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. The New England Journal of Medicine, 351, 13–22. 10.1056/NEJMoa040603

Hourani, L., Williams, J., Bray, R. M., Wilk, J. E., & Hoge, C. W. (2016). Gender differences in posttraumatic stress disorder and help seeking in the U.S. Army. Journal of Women’s Health, 25, 22–31. 10.1089/jwh.2014.5078

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. 10.1080/10705519909540118

Iversen, A. C., Fear, N. T., Ehlers, A., Hacker Hughes, J., Hull, L., Earnshaw, M., . . .Hotopf, M. (2008). Risk factors for post-traumatic stress disorder among UK Armed Forces personnel. Psychological Medicine, 38, 511–522. 10.1017/S0033291708002778

Jorm, A. F., & Wright, A. (2008). Influences on young people’s stigmatising attitudes towards peers with mental disorders: National survey of young Australians and their parents. The British Journal of Psychiatry, 192, 144–149. 10.1192/bjp.bp.107.039404

Judd, F., Jackson, H., Komiti, A., Murray, G., Fraser, C., Grieve, A., & Gomez, R. (2006). Help-seeking by rural residents for mental health problems: The importance of agrarian values. Australian and New Zealand Journal of Psychiatry, 40, 769–776. 10.1080/j.1440-1614.2006.01882.x

Julian, T. W., McKenry, P. C., & McKelvey, M. W. (1994). Cultural variations in parenting: Perceptions of Caucasian, African-American, Hispanic, and Asian-American parents. Family Relations, 43, 30–37. 10.2307/585139

Kaneshiro, B., Geling, O., Gellert, K., & Millar, L. (2011). The challenges of collecting data on race and ethnicity in a diverse, multiethnic state. Hawaii Medical Journal, 70, 168–171.

Kim, B. S., Atkinson, D. R., & Umemoto, D. (2001). Asian cultural values and the counseling process current knowledge and directions for future research. The Counseling Psychologist, 29, 570–603. 10.1177/0011000001294006

Kim, P. Y., Thomas, J. L., Wilk, J. E., Castro, C. A., & Hoge, C. W. (2010). Stigma, barriers to care, and use of mental health services among active duty and National Guard soldiers after combat. Psychiatric Services, 61, 582–588. 10.1176/ps.2010.61.6.582

Kuehner, C. A. (2013). My military: A navy nurse practitioner’s perspective on military culture and joining forces for veteran health. Journal of the American Association of Nurse Practitioners, 25, 77–83. 10.1111/j.1745-7599.2012.00810.x

Lam, C. S., Tsang, H., Chan, F., & Corrigan, P. W. (2006). Chinese and American perspectives on stigma. Rehabilitation Education, 20, 269–279.

Leaf, P. J., Bruce, M. L., & Tischler, G. L. (1986). The differential effect of attitudes on the use of mental health services. Social Psychiatry, 21, 187–192. 10.1007/BF00583999

Leong, F. T., & Lau, A. S. (2001). Barriers to providing effective mental health services to Asian Americans. Mental Health Services Research, 3, 201–214. 10.1023/A:1013177014788

Lepore, S. J., Silver, R. C., Wortman, C. B., & Wayment, H. A. (1996). Social constraints, intrusive thoughts, and depressive symptoms among bereaved mothers. Journal of Personality and Social Psychology, 70, 271–282. 10.1037/0022-3514.70.2.271

Lin, C. Y., & Tsai, M. C. (2016). Effects of family context on adolescents’ psychological problems: Moderated by pubertal timing, and mediated by self-esteem and interpersonal relationships. Applied Research in Quality of Life, 11, 907–923. 10.1007/s11482-015-9410-2

Lin, E., Goering, P., Offord, D. R., Campbell, D., & Boyle, M. H. (1996). The use of mental health services in Ontario: Epidemiologic findings. Canadian Journal of Psychiatry, 41, 572–577.

Link, B. G., Cullen, F. T., Struening, E., Shrout, P. E., & Dohrenwend, B. P. (1989). A modified labeling theory approach to mental disorders: An empirical assessment. American Sociological Review, 54, 400–423. 10.2307/2095613

Link, B. G., & Phelan, J. C. (2001). Conceptualizing stigma. Annual Review of Sociology, 27, 363–385. 10.1146/annurev.soc.27.1.363

Livingston, J. D., & Boyd, J. E. (2010). Correlates and consequences of internalized stigma for people living with mental illness: A systematic review and meta-analysis. Social Science & Medicine, 71, 2150–2161. 10.1016/j.socscimed.2010.09.030

Lubke, G. H., & Muthén, B. O. (2004). Applying multigroup confirmatory factor models for continuous outcomes to Likert scale data complicates meaningful group comparisons. Structural Equation Modeling, 11, 514–534. 10.1207/s15328007sem1104_2

Mackenzie, C. S., Gekoski, W. L., & Knox, V. J. (2006). Age, gender, and the underutilization of mental health services: The influence of help-seeking attitudes. Aging and Mental Health, 10, 574–582. 10.1080/13607860600641200

Mak, W. W., & Cheung, R. Y. (2008). Affiliate stigma among caregivers of people with intellectual disability or mental illness. Journal of Applied Research in Intellectual Disabilities, 21, 532–545. 10.1111/j.1468-3148.2008.00426.x

Mak, W. W., & Cheung, R. Y. (2010). Self-stigma among concealable minorities in Hong Kong: Conceptualization and unified measurement. American Journal of Orthopsychiatry, 80, 267–281. 10.1111/j.1939-0025.2010.01030.x

Mangione, T. W. (1998). Mail surveys. In L.Bickman & D. J.Rog (Eds.), Handbook of applied social research methods (pp. 399–427). Thousand Oaks, CA: Sage.

Moses, T. (2010). Being treated differently: Stigma experiences with family, peers, and school staff among adolescents with mental health disorders. Social Science & Medicine, 70, 985–993. 10.1016/j.socscimed.2009.12.022

Murray-Swank, A., Glynn, S., Cohen, A. N., Sherman, M., Medoff, D. P., Fang, L. J., . . .Dixon, L. B. (2007). Family contact, experience of family relationships, and views about family involvement in treatment among VA consumers with serious mental illness. Journal of Rehabilitation Research and Development, 44, 801–811. 10.1682/JRRD.2006.08.0092

Muthén, B., & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Mplus Web Notes, 4, 1–22.

Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.

Ojeda, V. D., & Bergstresser, S. M. (2008). Gender, race-ethnicity, and psychosocial barriers to mental health care: An examination of perceptions and attitudes among adults reporting unmet need. Journal of Health and Social Behavior, 49, 317–334. 10.1177/002214650804900306

Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241–286. 10.1006/ceps.2000.1040

Ortega, A. N., & Rosenheck, R. (2000). Posttraumatic stress disorder among Hispanic Vietnam veterans. The American Journal of Psychiatry, 157, 615–619. 10.1176/appi.ajp.157.4.615

O’Toole, B. I., Marshall, R. P., Schureck, R. J., & Dobson, M. (1999). Combat, dissociation, and posttraumatic stress disorder in Australian Vietnam veterans. Journal of Traumatic Stress, 12, 625–640. 10.1023/A:1024765001122

Ouimette, P., Vogt, D., Wade, M., Tirone, V., Greenbaum, M. A., Kimerling, R., . . .Rosen, C. S. (2011). Perceived barriers to care among Veterans Health Administration patients with posttraumatic stress disorder. Psychological Services, 8, 212–223. 10.1037/a0024360

Pietrzak, R. H., Johnson, D. C., Goldstein, M. B., Malley, J. C., & Southwick, S. M. (2009). Perceived stigma and barriers to mental health care utilization among OEF-OIF veterans. Psychiatric Services, 60, 1118–1122. 10.1176/ps.2009.60.8.1118

Ritsher, J. B., Otilingam, P. G., & Grajales, M. (2003). Internalized stigma of mental illness: Psychometric properties of a new measure. Psychiatry Research, 121, 31–49. 10.1016/j.psychres.2003.08.008

Sanchez, F., & Gaw, A. (2007). Mental health care of Filipino Americans. Psychiatric Services, 58, 810–815. 10.1176/ps.2007.58.6.810

Sandhu, D. S. (1997). Psychocultural profiles of Asian and Pacific Islander Americans: Implications for counseling and psychotherapy. Journal of Multicultural Counseling and Development, 25, 7–22. 10.1002/j.2161-1912.1997.tb00312.x

Scarpa, A., Haden, S. C., & Hurley, J. (2006). Community violence victimization and symptoms of posttraumatic stress disorder the moderating effects of coping and social support. Journal of Interpersonal Violence, 21, 446–469. 10.1177/0886260505285726

Schell, T. L., & Marshall, G. N. (2008). Survey of individuals previously deployed for OEF/OIF. In T.Tanielian & L. H.Jaycox (Eds.), Invisible wounds of war: Psychological and cognitive injuries, their consequences, and services to assist recovery (pp. 87–115). Santa Monica, CA: RAND Corporation.

Seal, K. H., Metzler, T. J., Gima, K. S., Bertenthal, D., Maguen, S., & Marmar, C. R. (2009). Trends and risk factors for mental health diagnoses among Iraq and Afghanistan veterans using Department of Veterans Affairs health care, 2002–2008. American Journal of Public Health, 99, 1651–1658. 10.2105/AJPH.2008.150284

Shrivastava, A., Johnston, M., De Sousa, A., Sonavane, S., & Shah, N. (2014). Psychiatric treatment as anti-stigma intervention: Objective assessment of stigma by families. International Journal of Medicine and Public Health, 4, 491–495. 10.4103/2230-8598.144126

Skopp, N. A., Bush, N. E., Vogel, D. L., Wade, N. G., Sirotin, A. P., McCann, R. A., & Metzger-Abamukong, M. J. (2012). Development and initial testing of a measure of public and self-stigma in the military. Journal of Clinical Psychology, 68, 1036–1047. 10.1002/jclp.21889

Spoont, M. R., Nelson, D. B., Murdoch, M., Rector, T., Sayer, N. A., Nugent, S., & Westermeyer, J. (2014). Impact of treatment beliefs and social network encouragement on initiation of care by VA service users with PTSD. Psychiatric Services, 65, 654–662. 10.1176/appi.ps.201200324

Stecker, T., Fortney, J. C., Hamilton, F., & Ajzen, I. (2007). An assessment of beliefs about mental health care among veterans who served in Iraq. Psychiatric Services, 58, 1358–1361. 10.1176/ps.2007.58.10.1358

Steenkamp, J. B. E., & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78–90. 10.1086/209528

Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173–180. 10.1207/s15327906mbr2502_4

Stewart, H., Jameson, J. P., & Curtin, L. (2015). The relationship between stigma and self-reported willingness to use mental health services among rural and urban older adults. Psychological Services, 12, 141. 10.1037/a0038651

Struening, E. L., Perlick, D. A., Link, B. G., Hellman, F., Herman, D., & Sirey, J. A. (2001). Stigma as a barrier to recovery: The extent to which caregivers believe most people devalue consumers and their families. Psychiatric Services, 52, 1633–1638. 10.1176/appi.ps.52.12.1633

Stuart, H. (2008). Fighting the stigma caused by mental disorders: Past perspectives, present activities, and future directions. World Psychiatry, 7, 185–188.

Su, C. T., Wang, J. D., & Lin, C. Y. (2013). Child-rated versus parent-rated quality of life of community-based obese children across gender and grade. Health and Quality of Life Outcomes, 11, 206. 10.1186/1477-7525-11-206

Sue, S., Fujino, D. C., Hu, L. T., Takeuchi, D. T., & Zane, N. W. (1991). Community mental health services for ethnic minority groups: A test of the cultural responsiveness hypothesis. Journal of Consulting and Clinical Psychology, 59, 533. 10.1037/0022-006X.59.4.533

Tay, L., Meade, A. W., & Cao, M. (2015). An overview and practical guide to IRT measurement equivalence analysis. Organizational Research Methods, 18, 3–46. 10.1177/1094428114553062

Tsai, J., & Radhakrishnan, R. (2012). Characterizing Asian American and Pacific Islander veterans who use health services. Military Medicine, 177, 1–3. 10.7205/MILMED-D-11-00276

U.S. Department of Health and Human Services. (2011). Final data collection standards for race, ethnicity, primary language, sex, and disability status required by Section 4302 of the Affordable Care Act. Washington, DC: Office of Management and Budget.

Van Brakel, W. H. (2006). Measuring health-related stigma—A literature review. Psychology, Health and Medicine, 11, 307–334. 10.1080/13548500600595160

Vogel, D. L., Wade, N. G., & Ascheman, P. L. (2009). Measuring perceptions of stigmatization by others for seeking psychological help: Reliability and validity of a new stigma scale with college students. Journal of Counseling Psychology, 56, 301–308. 10.1037/a0014903

Vogel, D. L., Wade, N. G., & Haake, S. (2006). Measuring the self-stigma associated with seeking psychological help. Journal of Counseling Psychology, 53, 325–337. 10.1037/0022-0167.53.3.325

Vogt, D. (2011). Mental health-related beliefs as a barrier to service use for military personnel and veterans: A review. Psychiatric Services, 62, 135–142. 10.1176/ps.62.2.pss6202_0135

Vogt, D., Fox, A. B., & Di Leone, B. A. (2014). Mental health beliefs and their relationship with treatment seeking among U.S. OEF/OIF veterans. Journal of Traumatic Stress, 27, 307–313. 10.1002/jts.21919

Wahl, O. F., & Harman, C. R. (1989). Family views of stigma. Schizophrenia Bulletin, 15, 131–139. 10.1093/schbul/15.1.131

West, A. N., Lee, R. E., Shambaugh-Miller, M. D., Bair, B. D., Mueller, K. J., Lilly, R. S., . . .Hawthorne, K. (2010). Defining “rural” for veterans’ health care planning. The Journal of Rural Health, 26, 301–309. 10.1111/j.1748-0361.2010.00298.x

Whealin, J., Nelson, D., Mahoney, M., & Kawasaki, M. (2016). A qualitative examination of needs and preferences of ethnoracially diverse rural Veterans with mental health problems. Manuscript in preparation.

Whealin, J. M., Sandoval de Lopez, T., Pietrzak, R. H., Yoneda, A., Woolaway-Bickel, K., & Southwick, S. M. (Eds.). (2012). Factors associated with OEF/OIF/OND Veterans’ engagement in psychotherapy for PTSD. Hauppauge, NY: Nova Science Publishers.

Yang, L. H., Phelan, J. C., & Link, B. G. (2008). Stigma and beliefs of efficacy towards traditional Chinese medicine and Western psychiatric treatment among Chinese Americans. Cultural Diversity & Ethnic Minority Psychology, 14, 10–18. 10.1037/1099-9809.14.1.10

Yoshioka, M. R., Gilbert, L., El-Bassel, N., & Baig-Amin, M. (2003). Social support and disclosure of abuse: Comparing South Asian, African American, and Hispanic battered women. Journal of Family Violence, 18, 171–180. 10.1023/A:1023568505682

Submitted: April 17, 2016 Revised: October 3, 2016 Accepted: October 4, 2016

A Meta-Intervention to Increase Completion of an HIV-Prevention Intervention: Results From a Randomized Controlled Trial in the State of Florida

Acknowledgement:

Retention and completion are critical components of the effectiveness of HIV-prevention interventions in real-world conditions and have established psychological determinants, such as attitudes and intentions (Albarracín, Durantini, Earl, Gunnoe, & Leeper, 2008; Noguchi, Albarracín, Durantini, & Glasman, 2007). Increasing retention is vital for public health because multisession behavioral interventions to reduce HIV risk are often more efficacious than single-session ones (Albarracín et al., 2005; Crepaz et al., 2014; Johnson et al., 2009; Meader et al., 2013). For example, the positive behavior change elicited by HIV-prevention interventions for clients of STI clinics is d = 0.33 for multisession programs, but only d = 0.06 for single-session programs (analyses of the data from Albarracín et al., 2005). However, when tested under conditions similar to the ones that are likely during actual implementation (e.g., lack of payments or other incentives), these multisession interventions show relatively low retention (Noguchi et al., 2007). Specifically, with the exception of interventions with captive audiences (e.g., prisons, inpatients), which show 100% completion rates, experimental interventions show a rate of completion of approximately 50% for participants initially enrolled (Branson, Peterman, Cannon, Ransom, & Zaidi, 1998; McMahon, Malow, Jennings, & Gómez, 2001). Without high retention, HIV-prevention interventions have less of an impact on behavior and clinical outcomes. Estimated associations for behavior change show that interventions with less than 50% retention rates produce a long-term decrease in HIV-safe behavior (d = −0.29), compared with an increase in HIV-safe behavior (d = 0.41) for those with 100% retention rates (Johnson et al., 2009). The present research examined the efficacy of two simple, postsession, messages to increase retention in a three-session risk-reduction counseling program. These messages were designed to either empower clients as agents responsible for their own change or highlight the instrumental outcomes of the intervention in terms of participants’ lives (e.g., addressing health concerns other than HIV, offering employment related information). The experimental design included five conditions, namely each of these messages, a combination of both, as well as two control conditions. The outcome variable was completion of a three-session counseling program.

Variability in Exposure to Behavioral Intentions

A number of interventions have been produced to change behaviors that place people at risk for HIV (Albarracín et al., 2005; Centers for Diseases Control and Prevention [CDC], 2007; Lorimer et al., 2013). These interventions are typically tested under conditions that ensure the validity of the outcome assessments (Cook & Campbell, 1979). Thus, researchers try to involve community members to see if a particular intervention works for them. Social networks are called upon to recruit these participants and numerous incentives and facilitators are used to ensure access to the desired sample of exposed participants, as well as low attrition (De Walque et al., 2012; Exner, Hoffman, Parikh, Leu, & Ehrhardt, 2002; Lauby et al., 1996; Linnan et al., 2002; Packel et al., 2012; Rabinowitz, 2002; Raj et al., 2001; Roffman, Picciano, Bolan, & Kalichman, 1997; Schilling & Sachs, 1993; Schweitzer, 1997; Tobias, Wood, & Drainoni, 2006). Although these procedures are necessary to determine whether a program works for an exposed population (efficacy trial), they remove the reluctance to participate when the intervention is implemented (Catania, Gibson, Chitwood, & Coates, 1990; Lauby et al., 1996; Packel et al., 2012). Contemporary research must thus address the fundamental scientific problem of variability in exposure to behavioral interventions, including completion of a program designed to elicit behavioral or medical change.

Despite the above-mentioned method of removing selection and attrition during tests of intervention efficacy, in real-world conditions, people choose to take part in preventive interventions (Albarracín et al., 2008; Condelli, Koch, & Fletcher, 2000; DiFranceisco et al., 1998; Hennessy, Mercier, Williams, & Arno, 2002; Noguchi et al., 2007; Rutledge, Roffman, Picciano, Kalichman, & Berghuis, 2002; Veach, Remley, Kippers, & Sorg, 2000; Wagenaar et al., 2012). Given limited time and interest, clients of health facilities can accept or refuse to take part in an HIV-prevention counseling session (Albarracín et al., 2008; Grady, Kegeles, Lund, Wolk, & Farber, 1983; Katz et al., 2015; Noguchi et al., 2007; Minder, Müller, Gillmann, Beck, & Stuck, 2002; Wilson & Albarracín, 2015). Moreover, some of the audiences most vulnerable to HIV are the least likely to complete HIV-prevention interventions (Earl et al., 2009; Liu et al., 2014; Noguchi et al., 2007; Wilson & Albarracín, 2015; Yancey, Ortega, & Kumanyika, 2006). In particular, frequent condom users are more likely to complete pro–condom-use interventions than infrequent ones (Earl et al., 2009; Noguchi et al., 2007). Thus, efficacious interventions may not reach the vulnerable audiences in need of interventions.

Given that interventions need to fully reach vulnerable audiences, not just willing ones, it is imperative to develop and test procedures that increase participation by these populations (Albarracín et al., 2008; Wilson & Albarracín, 2015). Procedures can be designed to change an audience’s behavior with respect to the preventive interventions themselves, including enrollment and retention. These procedures, termed meta-interventions, entail a standardized introduction or context change (e.g., delivery setting) intended to increase exposure to a behavioral intervention (Albarracín et al., 2008; Albarracín, Leeper, Earl, & Durantini, 2008; Wilson, Durantini, Albarracín, Crause, & Albarracín, 2013). In past research, participants with prior infrequent condom use were offered an HIV-counseling session using one of four scripted introductions to the program (Albarracín et al., 2008). A randomly assigned meta-intervention conveying that counseling participants are free not to change (empowering video) was more effective than other introductions (one promising change and another providing basic information about the counseling) or no introduction (just an offer to take part). Unobtrusive observers recorded the extent to which participants agreed to the counseling when asked, and also collected supplementary data on participants’ reading of brochures and viewing of videos. As hypothesized, the empowering meta-intervention produced high levels of enrollment in the counseling (Albarracín et al., 2008). In addition, viewing the video had an independent effect on enrollment, such that viewers of the video were more likely to enroll in counseling than nonviewers (Albarracín et al., 2008).

Selective Exposure to Interventions

Retention in HIV-prevention interventions can be understood as a form of selective exposure to information (Albarracín & Mitchell, 2004; Earl & Nisson, 2015; Noguchi et al., 2007). Selective exposure comprises biased information seeking behavior and was first studied by Festinger (1964; for reviews see Eagly & Chaiken, 1993; Freedman & Sears, 1965; Frey, 1986). Exposure to an intervention (in this case, staying in and completing a program) depends on two sets of motivations (i.e., goals or desired endstates; Lewin, 1926; see also Atkinson, 1964; McClelland, 1951; Nuttin, 1980; for other classifications of human motives, see Chaiken, Wood & Eagly, 1996; Eagly, 2007; Johnson & Eagly, 1989; Noguchi et al., 2007; Prislin & Wood, 2005; Wyer & Albarracín, 2005). On the one hand, individuals are motivated to achieve subjective self-validation, which comprises the defense of prior beliefs and practices in the domain of HIV prevention (Albarracín & Mitchell, 2004; Albarracín et al., 2008; Noguchi et al., 2007; see also Kunda, 1990; Molden & Higgins, 2005). On the other hand, individuals are motivated to maximize objective outcomes, such as reducing their risk for HIV and achieving other personal and emotional outcomes (Hart et al., 2009; Noguchi et al., 2007; Vanable et al., 2012).

A primary human motive is to achieve self-validation (Chaiken, Wood & Eagly, 1996; Eagly, 2007; Johnson & Eagly, 1989; Prislin & Wood, 2005; Wyer & Albarracín, 2005), and interventions may or may not fulfill it (Albarracín et al., 2008; Noguchi et al., 2007). Presumably due to the self-validation motive, individuals who engage in high-risk behavior are reluctant to enroll and stay in HIV-prevention interventions (Albarracín et al., 2008; Earl et al., 2009; Noguchi et al., 2007; Wilson & Albarracín, 2015).

Considering this, it is possible to design empowering messages to decrease defensiveness when recipients encounter a potential intervention that urges novel or even rejected practices (e.g., using condoms for nonusers; Albarracín et al., 2008). For example, past research has found an advantage in telling participants that change is up to them, that an intervention will simply open doors, and that they may or may not change if they participate. This type of meta-intervention puts recipients in a more active role by placing the burden of change upon them, while indirectly encouraging them to actively seek change (Amaro, 1995; Amaro & Raj, 2000; Albarracín et al., 2008; Freire, 1972; Higa, Marks, Crepaz, Liau, & Lyles, 2012; Putnam, 1911). Further, people are more likely to expose themselves to persuasive communications if they believe that they can resist their influence (Albarracín & Mitchell, 2004; Brehm, 1972; Brehm & Cohen, 1962; Watzlawick, 1978). For example, as infrequent condom users often do not want to use condoms (Albarracín, Johnson, Fishbein, & Muellerleile, 2001), highlighting the option of resistance increases their exposure to condom-use interventions (Albarracín et al., 2008). These processes have been investigated to improve enrollment in HIV programs (Albarracín et al., 2008), but not to achieve completion. As the dynamic of enrollment is similar to retention (Noguchi et al., 2007), similar messages may also increase retention in an HIV-prevention counseling program.

Besides self-validation, an important human motive is to maximize objective outcomes (Hart et al., 2009; Noguchi et al., 2007). Retention in an intervention is therefore likely to depend on the degree to which the intervention fulfills this motive (Albarracín et al., 2008; Earl et al., 2009; Noguchi et al., 2007; Vanable et al., 2012). For HIV-risk reduction interventions, objective outcomes include HIV-risk reduction (Floyd, Prentice-Dunn, & Rogers, 2000; Rosenstock, Strecher, & Becker, 1994), but also emotional and instrumental support (Durantini & Albarracín, 2009; Vanable et al., 2012). For people who engage in high-risk behavior, the risk-reduction outcome can conflict with the self-validation motive (Albarracín et al., 2008; Earl et al., 2009; Noguchi et al., 2007). Thus, emphasizing that the objective of an intervention is to change participants’ risk behavior can lead participants to reject the intervention and feel manipulated (Albarracín et al., 2008). In contrast, emphasizing the emotional, social and instrumental value of an intervention beyond HIV prevention can entice participation (Durantini & Albarracín, 2007, 2009; Liu et al., 2014). Past research supports this assertion, showing that, women seek out programs that provide social and emotional support (i.e., company, encouragement, and affection), whereas men seek out programs that provide instrumental support (i.e., health care or payments; Durantini & Albarracín, 2007, 2009). In this light, we tested the effect of messages that emphasize the emotional, instrumental, and (non-HIV related) physical health outcomes of returning to the sessions of an HIV-prevention and counseling program.

At the end of 2012, an estimated 1,218,400 people in the United States were living with HIV/AIDS (Hall et al., 2015). HIV incidence has remained relatively stable since the mid-1990s, with an estimated 50,000 persons becoming infected with HIV on any given year (Hall et al., 2008). Based on confidential-name–based HIV reports, 47,352 cases of HIV/AIDS were diagnosed in 35 U.S. areas (33 states, Guam, and the U.S. Virgin Islands) in 2013. Up to 2012, the cumulative number of individuals dead by HIV was 658,507, with Florida having one of the highest HIV disease death rates in the U.S. (Florida Department of Health [FDH], 2012). Also, in 2013, Florida ranked first in new infections per year (5,377 new infections) and second in number of cumulative reported HIV cases (49,058; CDC, 2013). In 2014, the largest estimated proportion of HIV/AIDS diagnoses in Florida was for men who have sex with men (MSM), and ethnic minority adults and adolescents infected through heterosexual contact (FDH, 2015). Clearly, protecting Floridians is a national health priority, particularly those from African American backgrounds, who are highly represented in our population.

In Florida, prevention is the most important tool to avoid an even more accelerated epidemic (FDH, 2007). Duval County (Electoral district 4) is an important area that has received relatively little research attention (compared with Dade County, e.g.). Considering sheer number of cases in 2014, Duval County, which includes Jacksonville, ranks 1st for Gonorrhea and 4th for Chlamydia (FDH, 2014a), and 4th for HIV (FDH, 2014b). Of 67 counties in the state of Florida, these rankings place the region at very high risk. Given these findings, ensuring intervention effectiveness for this population is key.

A randomized control trial was used to test the impact of video meta-interventions designed to either empower clients or remind them of the various objective goals fulfilled by the HIV-counseling program, and to compare these videos with control videos. The empowering meta-intervention entailed presenting the recipient as the motor of the behavior change (Albarracín et al., 2008). This strategy emphasized that the program could not change behavior unless the individual wanted it to. The instrumental video included descriptions of the sort of information and referrals the counselor could provide, in addition to giving information and guidance about HIV prevention. There was also a condition that combined the empowering and instrumental messages, as well as two control conditions. One control included stories about people living with HIV that were used in all experimental videos. The other control was more minimal, and simply presented educational information on reducing STIs. Thus, the design comprised five conditions to analyze their impact on completion of a CDC-recommended, three-session counseling program.

Our design also included a factor signaling whether perceptions of the video were measured. Although it was important to attempt to measure whether the videos induced expectations of empowerment and instrumental outcomes, including such blunt measures often alters the outcomes of experimental designs (Dholakia & Morwitz, 2002; Morwitz, Johnson, & Schmittlein, 1993). As a compromise, we randomized whether these measures appeared, and so only half of the sample completed these measures immediately after watching the meta-intervention video.

Enrollment

Clients from the STI clinics from the Florida Department of Health in Duval County were recruited (via flyers, referrals) for a study testing a three-session counseling program. To be eligible, individuals had to be between 18 and 35 years of age, report engaging in sexual activity in the past three months, and report using condoms never or occasionally. Participants were excluded if they were HIV-positive, or were trying to get pregnant or had a partner who was trying to get pregnant. Eligible participants were scheduled for their first study appointment. To ensure initial enrollment, participants were paid $35 for attending the first session, and $15 for subsequent sessions. The study was approved by the Institutional Review Boards (IRBs) of the University of Illinois, University of Pennsylvania, and Florida Department of Health, and each participant provided informed consent. Figure 1 describes all exclusions and Ns resulting from assignment procedures. The maximal control was by design smaller than the other conditions, including the minimal control. The trial was preregistered in clinicaltrials.gov (NCT01152281).

Figure 1. Recruitment and assignment.

Participants

Seven hundred twenty-two eligible participants (58% female) attended the initial counseling session, with a retention rate of 76% for the second session and a completion rate of 63% at the third session. The mean age of the sample was 26.54 (SD = 4.78). The majority of participants were African American (79%), and generally had an income under $9,999 per year (58%). Eighty-five percent reported having a main partner with whom they had a relationship on average of 2.37 years (SD = 2.10). Condom use in this sample was low, with only 1.1% reporting always using a condom when they had sex with their main partner. A full description of the same appears in Table 1.

Sample Description

The Counseling Intervention

The model of counseling that was used entailed a client-centered, cost-effective HIV-prevention program (CDC, 1993, 2007; Holtgrave, Valdiserri, Gerber, & Hinman, 1993; Kamb et al., 1998) facilitated by a counselor. This model’s efficacy has been demonstrated to significantly reduce STIs in a large multisite study (Kamb et al., 1998) and continues to be recommended as a standard for one-on-one counseling (CDC, 2007). The counseling seeks to reduce HIV risk behaviors by giving information, identifying risk behaviors, as well as steps to change them, and developing behavioral skills enabling safer behavior. This counseling can involve one or more sessions lasting at least 20 min, all of them following the same format. In our proposed study, a three-session model was used.

During the first session, participants received information regarding HIV transmission and prevention tailored to their culture, language, sex, gender, age, and educational level. The counselor ensured that the participant understood the information and that all of their misconceptions were corrected. The participant was encouraged to ask questions and clear their doubts. Following the informative part of the session, the counselor performed a personalized risk assessment, encouraging the participant to identify, understand, and acknowledge the behaviors and circumstances that put them at risk for being infected by HIV. Addressed topics included factors associated with risk behavior, such as using drugs or alcohol before sex, underestimating personal risk, having low self-efficacy, having distorted or fatalistic beliefs, and misperceiving peer norms. In addition, the counselor examined previous attempts made by the participant to reduce their risk and identified the reasons for their success or failure in these situations. This in-depth exploration allowed the counselor to help the participant consider ways to reduce personal risk and commit to a single, reachable step toward change. Once this risk assessment was complete, the counselor asked the participant to describe the risk-reduction step to be attempted (while acknowledging positive steps made), and helped the participant identify and commit to additional behavioral steps. Testing was also discussed, with referrals provided as needed.

During the following sessions, the counselor and the participant explored the success or failure of the steps proposed, and adjusted goals to the participant’s achievements. Furthermore, the second and third sessions also included a module for providing emotional support and addressing instrumental and/or medical concerns, in addition to HIV. This inclusion fulfilled the goal of providing supporting objective outcomes highlighted in some of the meta-intervention conditions. Among other things, after the HIV-risk reduction portion of the second and third counseling session was complete, the counselor discussed the physical and psychological symptoms, made referrals and provided information. This modification allowed us to test the effectiveness of messages that emphasized emotional and physical outcomes, with the counseling providing some venue for relief.

The counselors had good fidelity ratings using standard observation lists, and great high on cultural competency as measured with a valid and reliable questionnaire (Ponterotto, Alexander, & Grieger, 1995; Ponterotto, Potere, & Johansen, 2002). The counselors used written guides and records to ensure the use of a standardized procedure, and were closely supervised and retrained periodically. Therefore, after initial intensive training before the trial began, a check of videotaped sessions was performed to ensure proper application of the protocol. A random sample of 38 sessions showed 100% adherence to seven key dimensions of the protocol, which included appropriate introduction to the session, adequate performance of the risk assessment, proper evaluation of personal resources, proper evaluation of barriers, adequate integration of personal resources into newly set goals, and a clear closure. The average duration of the sessions was 25 min.

Meta-Intervention Messages

The messages were 24- to 34-min videos presented at the end of the first counseling session, to infer effects on retention at the second and third sessions. There were five videos, one for each condition, one resulting from crossing two meta-interventions, and two control videos.

The first experimental video, lasting 28 min, presented a meta-intervention conveying the message of being empowered (empowering condition). The video presented community members who talked about their experiences with HIV and counseling. This content was interspersed with messages delivered by these characters and professionals, conveying that subsequent counseling sessions were not intended to force change upon individuals. The stories were set at local places in North Florida (e.g., a fishing environment, a bar) with local music used in the background. The videos contained material in both Spanish and English, subtitled to the other language.

The second experimental video, lasting 26 min, presented a meta-intervention emphasizing the objective outcomes associated with HIV-prevention counseling (instrumental condition). This video presented the same stories as the first message. However, this experimental video also emphasized the emotional, social and objective (i.e., non-HIV/STI health) outcomes of returning to the counseling sessions. In this message, characters and professionals described how HIV-prevention counseling was also a venue to discuss personal problems, such as violence in the home or depression, and the extent to which many clients find emotional relief and social support from participating in the counseling. This message thus conveyed how HIV-prevention counseling often facilitates the treatment of other health problems, while also providing a venue for obtaining information about, and referral to, social services.

The third experimental video, lasting approximately 34 min, combined the first and the second meta-intervention messages. The final two videos were both control videos. The first, which lasted approximately 26 min, included the same stories and locations as those presented in the other three videos, but did not contain any of the meta-intervention messages (minimal control condition). The second, lasting 24 min, contained neither stories presented by local characters, nor any meta-intervention messages, but simply presented short vignettes aimed at increasing behavioral skills, perceived risk and knowledge about reducing STIs (maximal control condition; selected from video developed by Warner et al., 2008).

The use of two control conditions let us disentangle the effects of the meta-intervention messages from the community stories about HIV and counseling. Specifically, the difference between the minimal control video and those used in the three experimental conditions was the absence of a meta-intervention message; the rest of the content (e.g., the community stories) remained the same. The maximal control did not have either, and only presented risk and facts about STIs. Thus, it became possible to see whether differences in the three experimental conditions were attributable to the combination of the meta-intervention message and stories that were included, or were exclusive to the meta-intervention message. This condition was added after the project was funded and therefore had to be smaller because of funding constraints.

Design

This study crossed two meta-interventions: (a) a video message empowering or validating the client to return to the sessions, and (b) a video message highlighting opportunities for emotional and instrumental support (e.g., information about cardiovascular health, referrals etc.) facilitated by HIV-prevention counseling. The design had another factor, which concerned the inclusion of measures of expectation induced by the video, which were to appear immediately following the video. Only half of the sample completed these measures with the objective of avoiding measurement sensitivity. Thus, our design was a 2 (empowering meta-intervention: present vs. absent) × 2 (instrumental meta-intervention: present vs. absent) × 2 (measurement of mediators: present or absent) between-subjects factorial with the addition of a minimal control condition.

Baseline Measures

Data was collected using audio computer-assisted self-interview (ACASI) procedures. With this technique, participants listened to the question, while simultaneously reading them on the screen. ACASI procedures have been reported to increase accuracy with respect to non-normative behaviors and responses, thus decreasing the effects of social desirability and experimental demand (see, e.g., Des Jarlais et al., 1999; Mensch, Hewett, & Erulkar, 2003; Williams et al., 2000). Questionnaires were available in Spanish for participants who preferred it.

Baseline questionnaires measuring past behavior, intentions, and demographics were collected from participants before the start of their first counseling session. Questionnaire items were first transformed to a z-score, and then averaged, to produce a composite measure of condom, drug, alcohol and injection drug use, as well as a composite measure for number of sexual partners and intentions to use condoms.

Condom use and unprotected sex

Participants were asked about their condom use during intercourse in (a) the past month, (b) the past three months and (c) the past six months, as well as (d) how often they use condoms in general, (e) how many times they engaged in unprotected sex in the past six months, and (f) whether they used a condom the last time they had intercourse. These questions were asked in reference to participants’ main and other partner(s), and had acceptable internal consistency (α = .69 for main partner, and α = .63 for other partner).

Number of sexual partners

Participants were asked about the number of sexual partners they had in (a) the past 48 hours, (b) the past month, and (c) the past six months. This measure (National Institute of Drug Abuse [NIDA], 1991, 1993) had good internal consistency in our sample (α = .77; see also Edwards, Fisher, Johnson, Reynolds, & Redpath, 2007; Needle et al., 1995).

Alcohol use

Participants were asked to report their behaviors related to prior alcohol use. For those participants who reported that they drink alcohol, alcohol-use consisted of a single-item measure including reports of the number of times participants used alcohol during the past week.

Drug use

Participants were also asked to report their behaviors related to prior drug use. Drug-use measures included reports of the number of times participants used drugs (in general, as well as heroin, crack, and cocaine) during (a) the past 48 hours and (b) the past month. This measure had poor internal consistency in our sample (α = .52).

Injection drug use

Injection drug use was differentiated from the broader measure of drug use, as the level of HIV risk conferred by intravenous drug users is higher. Participants were asked (a) the number of times they injected drugs, (b) the frequency of sharing syringes, (c) the number of sharing partners, and (d) the number of times the equipment was sterilized between uses over a period of the past 48 hours, past month, and past six months. These questions were validated against HIV infection rates by Anthony et al. (1991), and had good internal consistency in our study (α = .85).

Intentions to use condoms

Participants were also asked about their intentions to use condoms, using previously validated measures (Albarracín et al., 2000; Earl et al., 2009; Kamb et al., 1998). Specifically, participants were asked how likely it was for them to use a condom with their partner (a) the next time they had intercourse, (b) every time for the next three months they had intercourse, and (c) every time for the next six months they had intercourse. Participants were also asked about (d) the strength of their intentions and (e) their motivation to use condoms with their partner in the next six months. These questions were asked in reference to participants’ main and other partner(s), and had excellent internal consistency (α = .94 for main partner, and α = .96 for other partner).

In addition to the above measures, participants were also asked standard items from the General Social Survey (http://gss.norc.org/) to assess structural variables, namely household income, level of education, race/ethnicity, and employment.

Measures of Video Acceptability, Counseling Expectations, and Return Intentions

The design included measures of the acceptability of the video, expectations of the following counseling sessions, and intentions to return. Measures were completed after the presentation of the video, by only half of the participants. Items for each measure were first transformed to a z-score, and then averaged, to create a composite measure for video acceptability, counseling expectations and return intentions.

To gauge video acceptability, participants were asked whether the video presented was (a) interesting, (b) useful, (c) enjoyable, (d) clear, and (e) relevant, as well as whether the video (f) made participants think, (g) taught them about condom use, and (h) presented new information. Participants were also asked whether the video made them (i) nervous, (j) worry, (k) feel compelled to do something they did not want to do, and (l) feel forced to change their beliefs or behaviors. Items (i) to (l) were first reverse scored, and then averaged with items (a) to (h). These measures had high internal consistency (α = .80).

Participants were also given measures of expectations about the counseling. Specifically, we asked participants whether they thought that the counseling would (a) force or (b) compel them to do things they did not like, (c) make them do things to please the counselor, (d) increase HIV safe behavior, (e) help them discuss health problems besides HIV and STIs, and (f) help them with their emotional concerns. Items (a) through (c) addressed empowerment expectations (α = .65), and items (d) through (f) addressed instrumental outcomes (α = .76).

Finally, participants were asked about the (a) strength of their return intentions and (b) how much they would enjoy returning (α = .67). All these measures were included as potential process data.

Completion Measure

Retention was observed during the last two sessions. When participants started the first session, the counselor indicated that the complete counseling program included two additional follow-up sessions. We measured retention by taking into account whether the participant completed all three sessions.

Across the board, there was a high completion rate of 63%, which is probably attributable to the use of payments for attendance at the return sessions. Before analyzing the outcome of the meta-intervention, we compared the demographic and behavioral profile of our sample. Any incidental difference was then controlled for in the main analysis.

Comparability Across Conditions

Although random assignment was intended to ensure comparability across conditions, we performed periodic checks to make sure there were no gender, age or race biases in the participant distribution. Table 1 provides a summary of relevant sample characteristics, by overall sample, as well as broken down by condition. One-way ANOVAs and chi-square tests revealed no significant difference in these variables across our five conditions (ps > .077), with the exception of age and race. Variability in race across conditions approached significance, χ(18) = 28.71, p = .052. There was a significant difference in the age of participants across conditions, F(9, 712) = 2.15, p = .024. A Tukey post hoc test revealed that participants’ age was significantly lower in the combined instrumental and empowering meta-intervention condition, when no measures of return expectations and intentions were included (M = 25.08, SD = 4.46), compared with the same condition presented when those variables were measured (M = 27.52, SD = 5.06). There were no significant differences in age across the other conditions (ps > .089).

Effects on Video Acceptability, Counseling Expectations, and Return Intentions

A multivariate analysis of variance, with our five meta-intervention message conditions as a factor, was conducted to analyze the impact of our experimental factors on video acceptability, counseling expectations (either empowering or instrumental) and the intention to return to counseling. Results revealed no significant effect of meta-intervention message (p = .14), indicating that our experimental factor did not affect participants’ reported acceptability of the video, empowering or instrumental expectations of counseling, or their intentions to return to the next counseling session. These findings suggest that any effect of the video either occurred outside of awareness, or could not be clearly reported by our participants on the scales that we developed. Means and standard deviations for video acceptability, counseling expectations and return intentions are presented in Table 2. The means in all cases were above the midpoints of the scales and suggest favorable perceptions of the video and a program perceived to be acceptable.

Means and Standard Deviations for Video Acceptability, Counseling Expectations, and Return Intentions Presented Across Meta-Intervention Message Conditions

Main Experimental Results

A logistic regression analysis with our three factors of empowering message, instrumental message, and mediator measurement presence, as well as all interactions, was conducted to analyze the impact of our experimental factors on counseling completion. In this analysis, the two control conditions were combined but a separate consideration of these two conditions does not alter our results. The analyses entailed a forward removal of predictors. The results from this analysis appear in Table 3. Results revealed a significant two-way interaction between the presentation of instrumental and empowering messages, as well as a significant interaction between mediator measure presence and presentation of instrumental messages (see Figure 2). Results indicated that the instrumental message alone was better than any of the other messages. Furthermore, the instrumental message was more effective than the empowering message in the absence of measures of mediating expectations. The overall gains in completion as a result of the instrumental message were 16%, suggesting success in the intended facilitation of counseling completion.

Final Results From Logistic Regression Analysis

Figure 2. Effects of meta-interventions.

This paper reported a large and complex randomized controlled trial testing meta-interventions to increase completion of a CDC-recommended counseling for HIV prevention. Our results identified a successful program—one that incorporates the counseling within a broader spectrum of goals that are likely salient to the clients of most programs to prevent, test for, and treat HIV. This finding is particularly impressive given that the completion rates in the sample were fairly high, probably attributable to a combination of excellent counseling technique and highly effective counselors, in addition to the use of payments for follow-up sessions. In other words, the room for improvement may have been limited to begin with, or at least, limited relative to the usually lower completion rates in the average comparable HIV-prevention trial (see Albarracín et al., 2005). Also, although the use of payments seemed desirable given pilot data showing lower completion than that ultimately obtained, hindsight suggests that the payments might have reduced the sensitivity of our completion measure.

The instrumental meta-intervention seemed important to test as it involves a patient centered approach to interventions (Lauver et al., 2002; Morgan & Yoder, 2012; Robinson, Callister, Berry, & Dearing, 2008) that is entirely consistent with psychological theories of persuasion and motivation. For a message to be well received, it is necessary for its content to be relevant to the audience and sufficiently consistent to ensure high level of message-consistent thinking and low levels of counterarguing (Albarracín, Johnson, & Zanna, 2005; Albarracín & Vargas, 2010). In the case of the instrumental message, highlighting the various personal goals that can be met through contact with the health system and associated services clearly retained participants who otherwise may have dropped out from the program. Future research should be conducted to replicate this finding in other areas, particularly HIV testing, HIV treatment, and introduction of pharmacological agents, such as PrEP.

Three aspects of our findings are noteworthy. First, the empowering meta-intervention, which had impressive results in a trial to increase acceptance of HIV-prevention counseling (Albarracín et al., 2008), did not yield improved completion. This result highlights that the determinants of enrollment and retention are different, with defensiveness playing a key role in initiation, but lack of relevance or perceived purpose probably underlying drop out. Second, the average completion rate was rather high and so our meta-intervention may have stronger effects when completion is low to begin. In our case, the high quality of the counseling and intensive training and supervision of the counselors, along with the payments, decreased the need for an intervention to ensure completion. Replications in conditions that are more conducive to higher drop out will therefore be highly informative. Third, as is common in testing behavioral interventions, the mediation analysis shed no light on the variables that led to the treatment outcome. It is of course possible that expectations did change, but participants did not have full introspective access to these contents due to the operation of relatively nonconscious processes. More likely, however, the questions were too involved and required a level of metacognition that is unfortunately not frequent for a sample with a low level of education. Perhaps a less directive assessment, such as a qualitative interview, may in the future increase understanding of the reasons underlying the success of the instrumental message.

Effects of the Mediators

The introduction of the mediators was expected to affect completion by sensitizing clients to the importance of completion. Often, calling attention to what the goals of a study are can introduce demands effects (Barabasz & Barabasz, 1992). An interesting study on measurement effects, however, was conducted by Glasman and colleagues (2015), who found that introducing measurements of risky behavior decreased the effect of a prevention intervention, suggesting a potential underestimation of the effect of behavioral programs. Both the demand and efficacy reduction patterns are entirely consistent with what we found. The inclusion of mediator measures increased completion, while also decreasing sensitivity to the meta-intervention. It seems likely that in-depth questions elicit cognitive and motivational processes, such as self-talk, that distract recipients from fully processing messages received immediately before (Glasman et al., 2015) or after (in our study) receiving a persuasive communication.

Remaining Questions and Limitations

There are several important questions to address, including possible differences in the intervention as a function of the delivered meta-intervention. For all clients, during the second and third counseling sessions, counselors discussed the physical and psychological symptoms associated with HIV, addressed instrumental and medical concerns, and provided emotional support for the participants regardless of the condition they were randomized to. Thus, concern over the influence of a meta-intervention condition on counselors’ interactions with participants is mitigated by the fact that the delivery of the counseling and the delivery of the meta-intervention messages were done by different team members. The counselor was blind to the meta-intervention condition, and so, all subsequent interactions with participants could not have been biased by knowledge of experimental condition.

With respect to generalizability to the population, we restricted the sample of participants to 18- to 35-year-olds because the estimated number of diagnoses of HIV infections in the U.S. is highest for this age range (Center for Disease Control, 2014). Additionally, prior work has shown that the mean age of participants enrolling in HIV-prevention intervention programs falls within this range (e.g., Liu et al., 2014; Wilson, Durantini, Albarracín, Crause, & Albarracín, 2013). Despite this age range restriction, we do not believe the generalizability of the studies should be affected, as the meta-interventions used target broad psychological themes, such as seeking self-validation or maximizing objective outcomes, which are not limited to specific age groups.

With respect to generalizability to the intervention format, a three-session counseling program was selected both because previous work we have done showed three-session interventions were a good length (Liu et al., 2014), and because of cost considerations. In principle, it is possible that the meta-intervention messages used in this study might have different efficacy with a longer program. However, given that completion was very high to begin, a program with lower rates of completion may show a stronger effect of the meta-interventions. Furthermore, we limited the presentation of the meta-intervention messages to participants who attended the first session as we were interested in the effect of these messages on completion of a program after it starts. Prior work we have conducted has already addressed the benefits of certain meta-intervention messages on increasing enrollment in HIV-prevention intervention programs (Albarracín et al., 2008), where it made more sense to present these messages to the entire sample at baseline.

One important consideration is the potential effect of the financial incentive to participation used in this study. As is well known, paying individuals for performing a task can reduce the perception of freedom of choice, and in turn decrease intrinsic motivation for the task (see, e.g., Festinger, 1964). In this context, payments could have led to lesser motivation to complete the program than lack of payments. This possibility seems unlikely because of the very high completion rates we obtained in our study. A likely possibility, however, is that the payment might have decreased motivation for the empowering condition, which emphasized freedom of choice. For example, emphasizing that clients are active participants and the motor of change may have reminded them of the payment and thus reduce their motivation to complete the program. Thus, future work should test the meta-intervention in the absence of payments.

It is important to further consider the effect of the meta-interventions, particularly the fact that the empowering one seemed to offer no benefits. In this regard, although empowering messages have been effective at eliciting enrollment before the intervention is delivered, a retention meta-intervention of this type may be directly in conflict with the obvious behavior-change intent of the program. The preventive nature of the program is likely to be apparent from a first session in which participants are encouraged to identify, understand and acknowledge the behaviors and circumstances that put them at risk for being infected with HIV. Also, it seems possible that the video might become impractical in some contexts, particularly if it were excessively long. However, the counseling session lasted 25 min in average, which along with a 26-min instrumental video would result in a first session of 51 min of length. Thus, it seems possible to integrate this meta-intervention into the regular operation of a clinic.

Although efficacious interventions are key tools in the prevention, detection, and treatment of HIV, the public health impact of these interventions are likely reduced when vulnerable populations do not complete the program. Our findings provide evidence that a meta-intervention simply describing HIV-prevention counseling as a venue where one can discuss personal problems or medical needs, and receive appropriate referrals to community resources, appears to be a promising strategy for increasing retention. The development of cost-effective tools to retain clients in multisession HIV-prevention programs could have a significant impact on the lives of those at greatest risk for HIV infection and may play a pivotal role in decreasing the number of new HIV infections in Florida, and in the Nation.

¹ Only one participant asked for the Spanish version of the measures.

Albarracín, D., Durantini, M. R., Earl, A., Gunnoe, J. B., & Leeper, J. (2008). Beyond the most willing audiences: A meta-intervention to increase exposure to HIV-prevention programs by vulnerable populations. Health Psychology, 27, 638–644. 10.1037/0278-6133.27.5.638

Albarracín, D., Gillette, J. C., Earl, A. N., Glasman, L. R., Durantini, M. R., & Ho, M. H. (2005). A test of major assumptions about behavior change: A comprehensive look at the effects of passive and active HIV-prevention interventions since the beginning of the epidemic. Psychological Bulletin, 131, 856–897.

Albarracín, D., Ho, R. M., McNatt, P. S., Williams, W. R., Rhodes, F., Malotte, C. K., . . . the Project RESPECT Study Group. (2000). Structure of outcome beliefs in condom use. Health Psychology, 19, 458–468. 10.1037/0278-6133.19.5.458

Albarracín, D., Johnson, B. T., Fishbein, M., & Muellerleile, P. A. (2001). Theories of reasoned action and planned behavior as models of condom use: A meta-analysis. Psychological Bulletin, 127, 142–161. 10.1037/0033-2909.127.1.142

Albarracín, D., Johnson, B. T., & Zanna, M. P. (2005). Attitudes: Introduction and scope. In D.Albarracín, B. T.Johnson, & M. P.Zanna (Eds.), The handbook of attitudes (pp. 3–20). Hillsdale, NJ: Erlbaum.

Albarracín, D., Leeper, J., Earl, A., & Durantini, M. R. (2008). From brochures to videos to counseling: Exposure to HIV-prevention programs. AIDS and Behavior, 12, 354–362. 10.1007/s10461-007-9320-x

Albarracín, D., & Mitchell, A. L. (2004). The role of defensive confidence in preference for proattitudinal information: How believing that one is strong can sometimes be a defensive weakness. Personality and Social Psychology Bulletin, 30, 1565–1584. 10.1177/0146167204271180

Albarracín, D., & Vargas, P. (2010). Attitudes and persuasion: From biology to social responses to persuasive intent. In S. T.Fiske, D. T.Gilbert, & G.Lindzey (Eds.), The handbook of social psychology (pp. 394–427). Hoboken, NJ: Wiley. 10.1002/9780470561119.socpsy001011

Amaro, H. (1995). Love, sex, and power. Considering women’s realities in HIV prevention. American Psychologist, 50, 437–447. 10.1037/0003-066X.50.6.437

Amaro, H., & Raj, A. (2000). On the margin: Power and women’s HIV risk reduction strategies. Sex Roles, 42, 723–749. 10.1023/A:1007059708789

Anthony, J. C., Vlahov, D., Celentano, D. D., Menon, A. S., Margolick, J. B., Cohn, S., . . .Polk, B. F. (1991). Self-report interview data for a study of HIV-1 infection among intravenous drug users: Description of methods and preliminary evidence on validity. Journal of Drug Issues, 21, 739–757. 10.1177/002204269102100405

Atkinson, J. W. (1964). An introduction to motivation. Princeton, NJ: D. Van Nostrand.

Barabasz, A. F., & Barabasz, M. (1992). Research designs and considerations. In E.Frornm & M. R.Nash (Eds.), Contemporary hypnosis research (pp. 173–200). New York, NY: Guilford Press. The preceding paper attributes the concept to Weber, S. J., & Cook, T. D. (1972). Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference. Psychological Bulletin, 77, 273–295.

Branson, B. M., Peterman, T. A., Cannon, R. O., Ransom, R., & Zaidi, A. A. (1998). Group counseling to prevent sexually transmitted disease and HIV: A randomized controlled trial. Sexually Transmitted Diseases, 25, 553–560. 10.1097/00007435-199811000-00011

Brehm, J. W. (1972). Responses to loss of freedom: A theory of psychological reactance. Morristown, NJ: General Learning Press.

Brehm, J. W., & Cohen, A. R. (1962). Explorations in cognitive dissonance. New York, NY: Wiley. 10.1037/11622-000

Catania, J. A., Gibson, D. R., Chitwood, D. D., & Coates, T. J. (1990). Methodological problems in AIDS behavioral research: Influences on measurement error and participation bias in studies of sexual behavior. Psychological Bulletin, 108, 339–362. 10.1037/0033-2909.108.3.339

Centers for Disease Control and Prevention. (2007). Revised guidelines for HIV counseling, testing, and referral. Morbidity Mortality Weekly Report, 50, 1–58.

Centers for Disease Control and Prevention. (2013). HIV surveillance report (Vol. 25). Atlanta, GA: U.S. Department of Health and Human Services.

Centers for Disease Control and Prevention. (2014). HIV surveillance report (Vol. 26). Atlanta, GA: U.S. Department of Health and Human Services.

Centers for Disease Control and Prevention. (1993). Update: Barrier protection against HIV infection and other sexually transmitted diseases. Morbidity Mortality Weekly Report, 42, 589–591, 597.

Chaiken, S., Wood, W., & Eagly, A. H. (1996). Principles of persuasion. In E. T.Higgins & A. W.Kruglanski (Eds.), Social psychology: Handbook of basic principles (pp. 702–742). New York, NY: Guilford Press.

Condelli, W. S., Koch, M. A., & Fletcher, B. (2000). Treatment refusal/attrition among adults randomly assigned to programs at a drug treatment campus: The New Jersey Substance Abuse Treatment Campus, Seacaucus, NJ. Journal of Substance Abuse Treatment, 18, 395–407. 10.1016/S0740-5472(99)00086-0

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin Company.

Crepaz, N., Tungol-Ashmon, M. V., Higa, D. H., Vosburgh, W., Mullins, M. M., Barham, T., . . .Lyles, C. M. (2014). A systematic review of interventions for reducing HIV risk behaviors among people living with HIV in the United States, 1988–2012. AIDS, 28, 633–656. 10.1097/QAD.0000000000000108

Des Jarlais, D. C., Paone, D., Milliken, J., Turner, C. F., Miller, H., Gribble, J., . . .Friedman, S. R. (1999). Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: A quasi-randomised trial. Lancet, 353, 1657–1661. 10.1016/S0140-6736(98)07026-3

de Walque, D., Dow, W. H., Nathan, R., Abdul, R., Abilahi, F., Gong, E., . . .Medlin, C. A. (2012). Incentivising safe sex: A randomised trial of conditional cash transfers for HIV and sexually transmitted infection prevention in rural Tanzania. British Medical Journal Open, 2, e000747. 10.1136/bmjopen-2011-000747

Dholakia, U. M., & Morwitz, V. G. (2002). The scope and persistence of mere-measurement effects: Evidence from a field study of consumer satisfaction measurement. Journal of Consumer Research, 29, 159–167. 10.1086/341568

DiFranceisco, W., Kelly, J. A., Sikkema, K. J., Somlai, A. M., Murphy, D. A., & Stevenson, L. Y. (1998). Differences between completers and early dropouts from 2 HIV intervention trials: A health belief approach to understanding prevention program attrition. American Journal of Public Health, 88, 1068–1073. 10.2105/AJPH.88.7.1068

Durantini, M. R., & Albarracín, D. (2007). Emotional and physical concerns as motives for seeking HIV prevention counseling. Unpublished manuscript.

Durantini, M. R., & Albarracín, D. (2009). Material and social incentives to participation in behavioral interventions: A meta-analysis of gender disparities in enrollment and retention in experimental human immunodeficiency virus prevention interventions. Health Psychology, 28, 631–640. 10.1037/a0015200

Eagly, A. H. (2007). In defense of ourselves: The effects of defensive processing on attitudinal phenomena. In M.Stroebe, J.de Wite, M.Hewstone, K.van den Bos, & H.Schut (Eds.), The scope of social psychology: Theory and applications (pp. 65–83). London, UK: Psychology Press.

Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes. Fort Worth, TX: Harcourt Brace Jovanovich.

Earl, A., Albarracín, D., Durantini, M. R., Gunnoe, J. B., Leeper, J., & Levitt, J. H. (2009). Participation in counseling programs: High-risk participants are reluctant to accept HIV-prevention counseling. Journal of Consulting and Clinical Psychology, 77, 668–679. 10.1037/a0015763

Earl, A., & Nisson, C. (2015). Applications of selective exposure and attention to information for understanding health and health disparities. Emerging Trends in the Social and Behavioral Sciences: An Interdisciplinary, Searchable, and Linkable Resource, 1–14. 10.1002/9781118900772.etrds0013

Edwards, J. W., Fisher, D. G., Johnson, M. E., Reynolds, G. L., & Redpath, D. P. (2007). Test-retest reliability of self-reported drug treatment variables. Journal of Substance Abuse Treatment, 33, 7–11. 10.1016/j.jsat.2006.11.007

Exner, T. M., Hoffman, S., Parikh, K., Leu, C. S., & Ehrhardt, A. (2002). HIV counseling and testing: Women’s experiences and the perceived role of testing as a prevention strategy. Perspectives on Sexual and Reproductive Health, 34, 76–83. 10.2307/3030210

Festinger, L. (1964). Conflict, decision, and dissonance. Stanford, CA: Stanford University Press.

Florida Department of Health. (2007). The Florida Division of Disease Control Surveillance Report. Tallahassee, FL: Author.

Florida Department of Health. (2012). HIV mortality in Florida in 2012. Tallahassee, FL: Bureau of Vital Statistics and Bureau of Communicable Diseases, HIV/AIDS Section.

Florida Department of Health. (2014a). Bureau of STD Prevention & Control. Retrieved from http://www.floridahealth.gov/diseases-and-conditions/disease-reporting-and-management/disease-reporting-and-surveillance/data-and-publications/_documents/2014-fl-msr.pdf#search=%222014%20gono%20chla%20bureau%20of%20std%20prevention%20%20%20control%20surveillance%22

Florida Department of Health. (2014b). Bureau of HIV/AIDS. Retrieved from http://www.floridahealth.gov/diseases-and-conditions/aids/surveillance/index.htmla

Florida Department of Health. (2015). HIV Disease: U.S. vs. Florida. Tallahassee, FL: Bureau of Vital Statistics and Bureau of Communicable Diseases, HIV/AIDS Section.

Floyd, D. L., Prentice-Dunn, S., & Rogers, R. W. (2000). A meta-analysis of research on protection motivation theory. Journal of Applied Social Psychology, 30, 407–429. 10.1111/j.1559-1816.2000.tb02323.x

Freedman, J. L., & Sears, D. O. (1965). Selective exposure. In L.Berkowitz (Ed.), Advances in experimental social psychology (Vol. 2, pp. 58–97). New York, NY: Academic Press.

Freire, P. (1972). Pedagogy of the oppressed. Harmondsworth, UK: Penguin.

Frey, D. (1986). Recent research on selective exposure to information. In L.Berkowitz (Ed.), Advances in experimental social psychology (Vol. 19, pp. 41–80). San Diego, CA: Academic Press. 10.1016/S0065-2601(08)60212-9

Glasman, L. R., Skinner, D., Bogart, L. M., Kalichman, S. C., McAuliffe, T., Sitzler, C. A., . . .Weinhardt, L. S. (2015). Do assessments of HIV risk behaviors change behaviors and prevention intervention efficacy? An experimental examination of the influence of type of assessment and risk perceptions. Annals of Behavioral Medicine, 49, 358–370. 10.1007/s12160-014-9659-y

Grady, K. E., Kegeles, S. S., Lund, A. K., Wolk, C. H., & Farber, N. J. (1983). Who volunteers for a breast self-examination program? Evaluating the bases for self-selection. Health Education Quarterly, 10, 79–94. 10.1177/109019818301000201

Hall, H. I., An, Q., Tang, T., Song, R., Chen, M., Green, T., . . . the Centers for Disease Control and Prevention (CDC). (2015). Prevalence of diagnosed and undiagnosed HIV infection—United States, 2008–2012. Morbidity Mortality Weekly Report, 64, 657–662.

Hall, H. I., Song, R., Rhodes, P., Prejean, J., An, Q., Lee, L. M., . . . the HIV Incidence Surveillance Group. (2008). Estimation of HIV incidence in the United States. JAMA: Journal of the American Medical Association, 300, 520–529. 10.1001/jama.300.5.520

Hart, W., Albarracín, D., Eagly, A. H., Brechan, I., Lindberg, M. J., & Merrill, L. (2009). Feeling validated versus being correct: A meta-analysis of selective exposure to information. Psychological Bulletin, 135, 555–588. 10.1037/a0015701

Hennessy, M., Mercier, M. M., Williams, S. P., & Arno, J. N. (2002). Client preferences for STD/HIV prevention programs. Evaluation and Program Planning, 25, 117–124. 10.1016/S0149-7189(02)00004-6

Higa, D. H., Marks, G., Crepaz, N., Liau, A., & Lyles, C. M. (2012). Interventions to improve retention in HIV primary care: A systematic review of U.S. studies. Current HIV/AIDS Reports, 9, 313–325. 10.1007/s11904-012-0136-6

Holtgrave, D. R., Valdiserri, R. O., Gerber, A. R., & Hinman, A. R. (1993). Human immunodeficiency virus counseling, testing, referral, and partner notification services. A cost-benefit analysis. Archives of Internal Medicine, 153, 1225–1230. 10.1001/archinte.1993.00410100057008

Johnson, B., & Eagly, A. H. (1989). The effects of involvement on persuasion: A meta-analysis. Psychological Bulletin, 106, 290–314. 10.1037/0033-2909.106.2.290

Johnson, B. T., Scott-Sheldon, L. A., Smoak, N. D., LaCroix, J. M., Anderson, J. R., & Carey, M. P. (2009). Behavioral interventions for African-Americans to reduce sexual risk of HIV: A meta-analysis of randomized controlled trials. Journal of Acquired Immune Deficiency Syndromes (1999), 51, 492. 10.1097/QAI.0b013e3181a28121

Kamb, M. L., Fishbein, M., Douglas, J. M., Jr., Rhodes, F., Rogers, J., Bolan, G., . . .for the Project RESPECT Study Group. (1998). Efficacy of risk-reduction counseling to prevent human immunodeficiency virus and sexually transmitted diseases: A randomized controlled trial. JAMA: Journal of the American Medical Association, 280, 1161–1167. 10.1001/jama.280.13.1161

Katz, I. T., Dietrich, J., Tshabalala, G., Essien, T., Rough, K., Wright, A. A., . . .Ware, N. C. (2015). Understanding treatment refusal among adults presenting for HIV-testing in Soweto, South Africa: A qualitative study. AIDS and Behavior, 19, 704–714. 10.1007/s10461-014-0920-y

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498. 10.1037/0033-2909.108.3.480

Lauby, J., Kotranski, L., Feighan, K., Collier, K., Semaan, S., & Halbert, J. (1996). Effects of intervention attrition and research attrition on the evaluation of an HIV prevention program. Journal of Drug Issues, 26, 663–677. 10.1177/002204269602600310

Lauver, D. R., Ward, S. E., Heidrich, S. M., Keller, M. L., Bowers, B. J., Brennan, P. F., . . .Wells, T. J. (2002). Patient-centered interventions. Research in Nursing & Health, 25, 246–255. 10.1002/nur.10044

Lewin, K. (1926). Intention, will and need. Psychologische Forschung, 7, 330–385. 10.1007/BF02424365

Linnan, L. A., Emmons, K. M., Klar, N., Fava, J. L., LaForge, R. G., & Abrams, D. B. (2002). Challenges to improving the impact of worksite cancer prevention programs: Comparing reach, enrollment, and attrition using active versus passive recruitment strategies. Annals of Behavioral Medicine, 24, 157–166. 10.1207/S15324796ABM2402_13

Liu, J., Jones, C., Wilson, K., Durantini, M. R., Livingood, W., & Albarracín, D. (2014). Motivational barriers to retention of at-risk young adults in HIV-prevention interventions: Perceived pressure and efficacy. AIDS Care, 26, 1242–1248. 10.1080/09540121.2014.896450

Lorimer, K., Kidd, L., Lawrence, M., McPherson, K., Cayless, S., & Cornish, F. (2013). Systematic review of reviews of behavioural HIV prevention interventions among men who have sex with men. AIDS Care, 25, 133–150. 10.1080/09540121.2012.699672

McClelland, D. C. (1951). Personality. New York, NY: William Sloane Associates. 10.1037/10790-000

McMahon, R. C., Malow, R. M., Jennings, T. E., & Gómez, C. J. (2001). Effects of a cognitive-behavioral HIV prevention intervention among HIV negative male substance abusers in VA residential treatment. AIDS Education and Prevention, 13, 91–107. 10.1521/aeap.13.1.91.18921

Meader, N., Semaan, S., Halton, M., Bhatti, H., Chan, M., Llewellyn, A., & Des Jarlais, D. C. (2013). An international systematic review and meta-analysis of multisession psychosocial interventions compared with educational or minimal interventions on the HIV sex risk behaviors of people who use drugs. AIDS and Behavior, 17, 1963–1978. 10.1007/s10461-012-0403-y

Mensch, B. S., Hewett, P. C., & Erulkar, A. S. (2003). The reporting of sensitive behavior by adolescents: A methodological experiment in Kenya. Demography, 40, 247–268. 10.1353/dem.2003.0017

Minder, C. E., Müller, T., Gillmann, G., Beck, J. C., & Stuck, A. E. (2002). Subgroups of refusers in a disability prevention trial in older adults: Baseline and follow-up analysis. American Journal of Public Health, 92, 445–450. 10.2105/AJPH.92.3.445

Molden, D. C., & Higgins, E. T. (2005). Motivated thinking. In K. J.Holyoak & R. G.Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp. 295–317). Cambridge, MA: Cambridge University Press.

Morgan, S., & Yoder, L. H. (2012). A concept analysis of person-centered care. Journal of Holistic Nursing, 30, 6–15. 10.1177/0898010111412189

Morwitz, V. G., Johnson, E., & Schmittlein, D. (1993). Does measuring intent change behavior?Journal of Consumer Research, 20, 46–61. 10.1086/209332

National Institute of Drug Abuse. (1991). Risk behavior assessment (October ed.). Rockville, MD: NIDA Community Research Branch.

National Institute of Drug Abuse. (1993). Risk behavior assessment (1st ed.). Ann Arbor, MI: Inter-University Consortium for Political and Social Research.

Needle, R., Fisher, D. G., Weatherby, N., Chitwood, D., Brown, B., Cesari, H., . . .Braunstein, M. (1995). Reliability of self-reported HIV risk behaviors of drug users. Psychology of Addictive Behaviors, 9, 242–250. 10.1037/0893-164X.9.4.242

Noguchi, K., Albarracín, D., Durantini, M. R., & Glasman, L. R. (2007). Who participates in which health promotion programs? A meta-analysis of motivations underlying enrollment and retention in HIV-prevention interventions. Psychological Bulletin, 133, 955–975. 10.1037/0033-2909.133.6.955

Nuttin, J. (1980). Motivation, planning and action. Leuven, Belgium: Leuven University Press.

Packel, L., Keller, A., Dow, W. H., de Walque, D., Nathan, R., & Mtenga, S. (2012). Evolving strategies, opportunistic implementation: HIV risk reduction in Tanzania in the context of an incentive-based HIV prevention intervention. PLoS ONE, 7, e44058. 10.1371/journal.pone.0044058

Ponterotto, J. G., Alexander, C. M., & Grieger, I. (1995). A multicultural competency checklist for counseling training programs. Journal of Multicultural Counseling and Development, 23, 11–20. 10.1002/j.2161-1912.1995.tb00262.x

Ponterotto, J. G., Potere, J. C., & Johansen, S. A. (2002). The Quick Discrimination Index: Normative data and user guidelines for counseling researchers. Journal of Multicultural Counseling and Development, 30, 192–207. 10.1002/j.2161-1912.2002.tb00491.x

Prislin, R., & Wood, W. (2005). Social influence: The role of social consensus in attitude and attitude change. In D.Albarracín, B. T.Johnson, & M. P.Zanna (Eds.), Handbook on attitudes and attitude change (pp. 671–706). Hillsdale, NJ: Erlbaum Inc.

Putnam, R. D. (1911). Making democracy work. Princeton, NJ: Princeton University Press.

Rabinowitz, D. C. (2002). Predictors of attrition in a predominantly Caucasian middle-class clinic-based weight loss program. Dissertation Abstracts International: Section B: The Sciences and Engineering, 62, 5976.

Raj, A., Amaro, H., Cranston, K., Martin, B., Cabral, H., Navarro, A., & Conron, K. (2001). Is a general women’s health promotion program as effective as an HIV-intensive prevention program in reducing HIV risk among Hispanic women?Public Health Reports, 116, 599–607. 10.1016/S0033-3549(04)50093-6

Robinson, J. H., Callister, L. C., Berry, J. A., & Dearing, K. A. (2008). Patient-centered care and adherence: Definitions and applications to improve outcomes. Journal of the American Academy of Nurse Practitioners, 20, 600–607. 10.1111/j.1745-7599.2008.00360.x

Roffman, R. A., Picciano, J. F., Bolan, M., & Kalichman, S. C. (1997). Factors associated with attrition from an HIV-prevention program for gay and bisexual males. AIDS and Behavior, 1, 125–135. 10.1023/B:AIBE.0000002973.21222.bc

Rosenstock, I. M., Strecher, V. J., & Becker, M. H. (1994). The health belief model and HIV risk behavior change. In R. J.DiClemente & J. L.Peterson (Eds.), Preventing AIDS: Theories and methods of behavioral interventions (pp. 5–24). New York, NY: Springer Science & Business Media. 10.1007/978-1-4899-1193-3_2

Rutledge, S. E., Roffman, R. A., Picciano, J. F., Kalichman, S. C., & Berghuis, J. P. (2002). HIV prevention and attrition: Challenges and opportunities. AIDS and Behavior, 6, 69–82. 10.1023/A:1014580629524

Schilling, R. F., & Sachs, C. (1993). Attrition from an evening alcohol rehabilitation program. The American Journal of Drug and Alcohol Abuse, 19, 239–248. 10.3109/00952999309002683

Schweitzer, C. N. (1997). A study of the relationship between psychosocial factors and attrition, compliance, and gains in Phase II cardiac rehabilitation. Dissertation Abstracts International: Section A: Humanities and Social Sciences, 58, 0392.

Tobias, C., Wood, S., & Drainoni, M. L. (2006). Ryan White Title I survey: Services for HIV-positive substance users. AIDS Patient Care and STDs, 20, 58–67. 10.1089/apc.2006.20.58

Vanable, P. A., Carey, M. P., Brown, J. L., Littlewood, R. A., Bostwick, R., & Blair, D. (2012). What HIV-positive MSM want from sexual risk reduction interventions: Findings from a qualitative study. AIDS and Behavior, 16, 554–563. 10.1007/s10461-011-0047-3

Veach, L. J., Remley, T. P., Jr., Kippers, S. M., & Sorg, J. D. (2000). Retention predictors related to intensive outpatient programs for substance use disorders. The American Journal of Drug and Alcohol Abuse, 26, 417–428. 10.1081/ADA-100100253

Wagenaar, B. H., Christiansen-Lindquist, L., Khosropour, C., Salazar, L. F., Benbow, N., Prachand, N., . . .Sullivan, P. S. (2012). Willingness of US men who have sex with men (MSM) to participate in Couples HIV Voluntary Counseling and Testing (CVCT). PLoS ONE, 7, e42953. 10.1371/journal.pone.0042953

Warner, L., Klausner, J. D., Rietmeijer, C. A., Malotte, C. K., O’Donnell, L., Margolis, A. D., . . . the Safe in the City Study Group. (2008). Effect of a brief video intervention on incident infection among patients attending sexually transmitted disease clinics. PLoS Medicine, 5, e135. 10.1371/journal.pmed.0050135

Watzlawick, P. (1978). The language of change: Elements of therapeutic communication. New York, NY: Norton & Company Inc.

Williams, M. L., Freeman, R. C., Bowen, A. M., Zhao, Z., Elwood, W. N., Gordon, C., . . .Signes, C. A. (2000). A comparison of the reliability of self-reported drug use and sexual behaviors using computer-assisted versus face-to-face interviewing. AIDS Education and Prevention, 12, 199–213.

Wilson, K., & Albarracín, D. (2015). Barriers to accessing HIV-prevention in clinic settings: Higher alcohol use and more sex partners predict decreased exposure to HIV-prevention counseling. Psychology, Health and Medicine, 20, 87–96. 10.1080/13548506.2014.902484

Wilson, K., Durantini, M. R., Albarracín, J., Crause, C., & Albarracín, D. (2013). Reducing cultural and psychological barriers to Latino enrollment in HIV-prevention counseling: Initial data on an enrollment meta-intervention. AIDS Care, 25, 881–887. 10.1080/09540121.2012.729803

Wyer, R. S., & Albarracín, D. (2005). The origins and structure of beliefs and goals. In D.Albarracín, B. T.Johnson, & M. P.Zanna (Eds.), Handbook of attitudes (pp. 273–322). Hillsdale, NJ: Erlbaum.

Yancey, A. K., Ortega, A. N., & Kumanyika, S. K. (2006). Effective recruitment and retention of minority research participants. Annual Review of Public Health, 27, 1–28. 10.1146/annurev.publhealth.27.021405.102113

Submitted: November 3, 2015 Revised: May 9, 2016 Accepted: July 4, 2016

A Method for Making Inferences in Network Analysis: Comment on Forbes, Wright, Markon, and Krueger (2017)

Acknowledgement: Douglas Steinley and Kenneth J. Sher were partially supported by National Institutes of Health Grant R01AA023248-01.

We commend the authors—Forbes, Wright, Markon, and Krueger (2017)—for bringing to light one of the issues that has arisen recently when attempting to make inferences about the structure of so-called psychopathology networks. Specifically, Forbes et al. note that there are emerging regarding our ability to determine the extent that network structures obtained from two (or more) samples are similar or different in key respects. That is, do they replicate? Judging the similarity of various symptom network solutions can be challenging for a number of reasons, not the least of which is the number of parameters modeled including the number of edges (paths) connecting nodes (symptoms) and various measures that are derived from these, perhaps most critically, centrality measures that speak to the relative importance of individual symptoms vis a vis the total network. Depending upon the specific measures of centrality under consideration, such centrality measures could represent a powerful way to identify symptoms that are more fundamental (as opposed to accessory) to a given syndrome, bridge distinct symptom networks, or strongly related to other symptoms.

However, to have confidence in the results of any single network estimation procedure, it is important to be able to demonstrate its replicability, that is how similar are the estimated edge weights and centrality parameters across two different samples. Problems in establishing “reproducibility” is an increasing focus of concern in both psychology (e.g., Pashler & Wagenmakers, 2012) and science more generally (e.g., Baker, 2016). As noted above, the sheer number of parameters estimated make establishing the similarity of two more networks challenging and traditional approaches such as intuitive “eye balling” of the magnitude of various measures of agreement, concordance, or replicability (for instance, the Pearson correlation coefficient, the intraclass correlation)—many of which will not be appropriate—fails to provide a framework for making strong inferential statements regarding replicability of network parameterizations. Moreover, for technical reasons, measures of association based on binary data provide special challenges owing to their sensitivity to base rates (i.e., marginal frequencies) which may differ across samples either by design or sampling effects. Having a methodology for assessing network similarity that adequately addresses this challenge would make it possible for researchers to validly judge reproducibility and replicability.

In this commentary, we describe a simple yet effective method for generating empirical confidence intervals for whatever statistic is desired for inspection and demonstrate its use on a sample data set under the conditions described by Forbes et al. (2017). We then go on to provide some general comments regarding its application.

More specifically, we first review the notion of empirically creating appropriate null distributions for quantities of interest. We then discuss the application of such approaches to traditional one-mode networks where one set of nodes that are similar to each other or the same type (e.g., individuals). Following this, we provide an overview of the structure of two-mode matrices as commonly thought of in terms of network structure or graph theory, where there are two different types of nodes (e.g., symptoms and people). After that, we introduce the proposed hypothesis testing for two-mode binary networks and explicate the method by reanalyzing one of the data sets from Forbes et al. (2017) to obtain confidence intervals under an appropriate null distribution for edges and centrality statistics. This allows us to conduct subsequent statistical tests by determining whether the statistic of interest lies within the null confidence interval. If it does, then we conclude that the observed network statistic is performing the same as it would under a random two-mode network; if it does not, then a process of potential interest is likely driving the observed value for the tested statistic.

In multivariate statistics, it is often difficult to derive the exact sampling distribution for many quantities of interest. A standard practice is to generate random data with appropriate constraints, fit the hypothesized model to these generated data, and then compare the quantity of interest when computed under random data with that in the observed network. If they are very similar, then we cannot conclude that the observed data are any different than random data. Such an approach is often employed in other areas of multivariate statistics (e.g., use of parallel analysis in factor analysis, Horn, 1965).

In parallel analysis, the eigenvalues obtained from the covariance (or correlation) matrix derived from the sample data are compared with the eigenvalues obtained from covariance (or correlation) matrices arising from completely random data. The central idea behind of parallel analysis is that if a set of measured variables arises from a common factor model with f factors, then the f largest eigenvalues as computed from the data set should exceed that of the expected (e.g., average) eigenvalues of random data. One critical factor in making this comparison is how the random data are generated. Specifically, the constraints that are placed on each of the data sets in the comparison population are critical. If the eigen-decomposition is on the correlation matrix, a common constraint is the generated data should have the same sample size (e.g., the same number of observations) and the same number of variables. If the eigen-decomposition is on the covariance matrix, an additional constraint for consideration would be to ensure that the randomly generated variables had the same variances.

Similarly, corresponding procedures for comparing observed fit statistics to fit statistics derived from random data has been implemented in the domain of cluster analysis. Specifically, Steinley (2006, 2007, 2008) and Steinley and Brusco (2011) generated distributions based on random data to test the quality of a cluster solution and to determine whether the correct number of clusters had been chosen, respectively. Using this general approach, Steinley (2004) developed a method for sampling cluster agreement-matrices to approximate the statistic’s sampling distribution. Although used extensively in multivariate statistics, such approaches have also seen use in traditional network analyses—as explained in the following section.

One-Mode Networks

For one-mode networks (e.g., the rows and the columns of the adjacency matrix describe the same entities, such as a friendship network) statistical inference has long been an active area of research and methodological development because the standard inferential techniques relying on independence of observations does not apply. As a solution, it has been argued that the testing network statistics of interest can be obtained by generating a null distribution from a population of networks that have an appropriate structure that the researcher wishes to control. Generally, these networks are generated uniformly (e.g., each network has an equal chance of being selected and included in the population. Anderson, Butts, and Carley (1999) described an approach to conduct hypothesis testing on network level statistics when controlling for the number of observations (e.g., the number of actors or nodes) and the overall density (e.g., the number of links). Additional constraints can be added to the generation process, such as controlling for the number of mutual, asymmetric, and null ties. Furthermore, for one-mode networks, the notion of generating networks with specific properties has flourished with recent developments in exponential random graph modeling (see Hunter & Handcock, 2006).

Two-Mode Matrices

Two-mode data binary data has long been of interest in the psychological sciences (Arabie & Hubert, 1990; Arabie, Hubert, & Schleutermann, 1990; Brusco & Steinley, 2006, 2007a, 2007b, 2009, 2011; Brusco, Shireman, & Steinley, in press; Rosenberg, Van Mechelen, & De Boeck, 1996). In terms of networks and graph theory, two-mode binary data can be thought of as a bipartite graph and represented in a standard data matrix with n rows and p columns (where, often, rows represent the subjects and columns represent the variables). In the network literature, these are often deemed affiliation matrices. Until recently, there has been a dearth of methods for generating binary affiliation matrices with fixed margins, with the ones being used somewhat cumbersome and inefficient and too slow for application to large networks (Admiraal & Handcock, 2008; Chen, Diaconis, Holmes, & Liu, 2005). However, within the last few years, a pair of papers have appeared (Harrison & Miller, 2013; Miller & Harrison, 2013) that provide a computationally efficient method for generating affiliation matrices with fixed margins.

Interestingly, the motivating example for this work arises out of ecology to test cohabitation of species in biogeographical data, where the data are often of the form Species × Habitat. As a bit of background, original work by Diamond (1975) made extensive claims about “species assembly rules” that aimed to explain the interaction between specific species and their chosen habitats—indeed, some of the relationships between species and habitats were quite strong. However, Connor and Simberloff (1979) argued that, if one controlled for the base rates of the different species and the different habitats, then many of these so-called “strong” associations could have arisen by random chance. According to Miller and Harrison (2013), this set off years of argument about the nature of null hypothesis testing and whether to control for the marginal distributions or not. Owing to the fact that the effect disappears when the marginal distributions are controlled for, the current guidelines recommend that the marginal distributions are fixed such that observed effects are not due to the base rates of the rows and columns alone.

If the two-mode matrix is viewed as a contingency table, then fixing both marginal distributions corresponds to Fisher’s exact test, with the following assumptions:

1. Each observation is classified into one and only one category of the row variable and into one and only one category of the column variable.
2. The N observations come from a random sample such that each observation has the same probability of being classified into the ith row and the jth column as any other observation.
3. The null hypothesis is: The event of an observation being in a particular row is independent of that same observation being in a particular column.

This framework will serve to generate the random matrices for our test, as described below.

The goal of this commentary is to present such a test based on generating a set(s) of matrices from prespecified marginal distributions. The desire is to provide a statistical context for evaluating the magnitude some measures of correspondence with respect to the expected parameter estimates generated from random processes with known base rates. Further, much like the ecology literature, we argue that in the psychopathology literature it is imperative to compare observed results to null distributions that have the same prevalences of the symptoms themselves and the same distributions of severity across the individuals. This is a necessary condition to begin making precise, generalizable statements about specific symptoms by guaranteeing that any observed/interpreted relationships between the variables is attributable to their co-occurrence within the same individuals are not an artifact imposed by the marginal distributions.

Holding the within-person marginal distributions constant is necessary because the goal of network analysis in some ways is much more lofty that that of the common cause models, such as factor analysis. In the latter, it is often assumed that all items loading onto the same factor should be (theoretically) exchangeable; however, in network analysis the goal is to discover the explicit connections between individual items—as such, the items themselves take on a much more pronounced role in the modeling process. Specifically, for factor models, knowing that a subject endorsed 3 of 5 items on a disorder is often enough; in fact, that is what much of the diagnostic literature is predicated on. Conversely, for network models, the specific 3 items are much more important because the goal is to make causal (or pseudocausal) connections between specific items. The bar is much higher. Likewise, holding the within-item prevalences constant is extremely important as well. Specifically, it can easily be shown that items that have a higher prevalence of occurring are more likely to have observed edges in the network model. Consequently, it is important to model random items with the same rates of occurrence to ensure that observed network edges are not merely a byproduct of items that have been endorsed more than other items. Without such a reference, researchers will remain in the dark about what their findings indicate with these newer network models. The following two subsections describe how the procedure works, with a more mathematical explanation provided in the Appendix.

Algorithm for Within Network Importance

The ingredients for testing so-called “within” network importance are fairly straightforward. Namely, all that is required is the original data matrix Xn×p with n rows and p columns as well as the estimated network statistic of interest, say θ̂. The goal is then to obtain the sampling distribution for θ under the null distribution that the location of the ones (i.e., the presence of the symptom, say) in the binary matrix X - marginal distributions for both the rows and columns.

Procedurally, the process is fairly simple. First, the user must choose the number of random data matrices to generate. After this is chosen, each random matrix is generated such that it has identical column totals (e.g., each item prevalence is the same in the random matrices as in the observed matrix) and row totals (e.g., the severity profiles of the observations in the random matrices are the same as in the observed matrix). However, in the random generation process, the elements in the random matrices are generated such that the rows and columns are independent of each other conditional on the row totals and column totals (e.g., the marginal distributions) being the same. After all of the random matrices are generated, the statistic of interest (e.g., edge weights, centrality parameters, etc.) is computed on each of the random matrices. Finally, the statistic for the observed data is compared with the reference distribution derived from the random matrices, allowing for the calculation of a percentile score. If this score is between the 2.5 percentile and the 97.5 percentile, we can say that the observed data are providing results that are consistent with what we would expect to see in random data. That is, the p values associated with the estimates do not exceed a nominal level of.05.

Algorithm for Between Importance

The first algorithm discusses determining whether the observed data deviates from the what would be observed via random chance and concerns network elements that are internal to one data set (hence, the moniker of “within”). However, it is also possible to compute the correspondence or stability of network statistics across different data sets, which we will term as “between.” To compute the correspondence between two networks, in additional data set, Yn2×p, is required (note that while the number of variables must be the same, the two data sets may have different numbers of observations). Network structures are then derived from both X and Y, and a measure of correspondence is computed. Then, pairs of random matrices are generated, with one corresponding to X and one corresponding to Y, and the respective marginal distributions are fixed to those of the observed matrices. The same measure of correspondence is computed for each pair for random matrices, allowing for the generation of sampling distribution under random chance to be obtained. The computation of percentile scores than proceeds in the same fashion as described for the within network importance.

Relationship to Bootstrapping

Recently, Epskamp, Borsboom, and Fried (in press) have introduced methods for applying bootstrap techniques to psychological networks. These methods are designed to (a) assess the accuracy of estimated network connections, (b) investigate the stability of centrality indices, and (c) test whether network connections and centrality estimates for different variables differ from each other. The primary difference between the methodology proposed above and bootstrapping approach is that the former is assessing whether the observed effects are different than what we would expect from random chance while the latter assess stability and accuracy of coefficient estimation.

As such, it is possible, and as we see in the examples below, perhaps expected (because of the constraints placed on the network space due to the marginal distributions) that estimated network effects can be both simultaneously stable and not different than random chance. Additionally, a confidence interval around the either the connections between variables or the centrality estimates, as derived from the bootstrap approach, cannot include zero and still overlap with (or even fall within) the confidence interval indicating what we would expect by chance alone. In that instance, the estimate would be stable, but not particularly interesting/informative—and, in such a scenario we would caution against interpreting the estimate because, while the effect is shown to exist, it is not different than what would be expected by chance alone. We summarize the four possible outcomes derived from the outcomes of the two testing procedures in Table 1.

Four Potential Outcomes Between Bootstrap Procedure and Random Chance

In Table 1, we provide the more general possibility of testing hypotheses (or constructing confidence interval) concerned with comparing the effects with a prespecified null value, v0. Likewise, we can construct, from the procedure outlined above, the expected value of the effect under random chance, vrc. Then, the combined information of whether the estimate (v̂) is different than the hypothesized value (v0) and/or the value under random chance (vrc) is useful for determining the interestingness/importance of an effect. Obviously, the most interesting effect will be situations where it is shown that the estimate is both different from the hypothesized value and what is expected under random chance. Conversely, if effects are not different from what is expected under random chance, the information derived from the bootstrap method is uninteresting regardless of the degree of stability, accuracy, or difference from the hypothesized value. Lastly, when the effects are different from chance but not different from the hypothesized value, the result could be potentially interesting. In terms of the network diagram, this would correspond to unobserved links between pairs of variables. As such, each observed link can be interesting (different from zero, different from random chance) or uninteresting (not different from random chance); likewise, each unobserved link could be uninteresting (not different from zero, not different from random chance) or potentially interesting (not different from zero, different from random chance). We use “potentially” interesting because relationships (e.g., presence of links) and conditional independence (e.g., absence of links) are both inferred when the network diagram is visually inspected.

The goal of these analyses is to infer the true underlying causal relationships between psychological symptoms, thus a significant result would be one that informs us of relations above and beyond what their prevalence rates tell us. Given the length constraints of the commentary, we focus on demonstrating the within-network evaluation algorithm for the NCS-R data as fit with the Ising model. Specifically, we will demonstrate the construction of confidence intervals around the estimated edge weights and centrality statistics as estimated using the qgraph package in the R statistical computing environment. To begin with, we need the marginal distributions of the NCS-R data. The frequency of each symptom is provided in Table 2, while the severity frequency (e.g., the number of symptoms endorsed by each subject, ranging from 0–18) is provided in Figure 1. From these data, we can immediately see the issue with only fixing one of the distributions. For instance, if we only fixed the prevalence of each symptom, then they would be distributed equally across the subjects, resulting in the average subject being assigned 3.75 symptoms. This would completely ignore the role of the severity continuum and its distribution in forming the network structure. Likewise, if we fixed only the severity marginals, we would see equally distributed counts over all of the symptoms, resulting in each having the same expected prevalence; consequently, differing prevalence rates alone could potentially define the “network” structure.

Frequency of Symptoms in the NCS-R Data

Figure 1. Frequency distribution of number of symptoms in NCS-R data showing the number of individuals with different numbers of symptoms.

Centrality Statistics

We implemented the algorithm as described above. For this small example, we generated 1000 matrices with marginals fixed to the values indicated in Table 2 and Figure 1. For each parameter of interest, we sorted the 1000 values from highest to lowest and created an empirical 99% confidence interval by taking the 5th and 995th ordered value. While we provide 99% confidence intervals here, more discussion is provided at the end of this section regarding choosing appropriate confidence levels. The confidence intervals for the centrality statistics are displayed in Table 3. From Table 3, we see that there are some variables that have significant centrality statistics when compared with what would be expected from chance. Significant values of centrality statistics are noted with an asterisk. For betweenness, we see that not all nonzero centrality statistics are significant, the most egregious being the betweenness score for “Anxiety about > 1 Event” is 38; however, we find that is not significantly different than would be expected by chance. This indicates that the magnitude of the centrality statistics does not necessarily indicate its relevance. We see similar results for closeness centrality, where 33% of the statistics are significant; unfortunately, without a formal test, it is impossible to determine which are different than chance as all of the estimates are within.017 of each other and magnitude alone does not confer significance. For strength, about two thirds of the centrality statistics are significant.

Confidence Intervals for Centrality Statistics

Finally, it is important to note that sometimes the centrality statistics are significantly less than would be expected by chance. This is true for “Chronic Anxiety” for betweenness and closeness, as well as “Muscle Tensions” for strength. This complicates the evaluation of this type of output because it is unknown whether observed results are significantly greater than chance, significantly less than chance, or no different than chance. Once again, the necessity for a formal test is highlighted.

Edge Weights

The difficulty is exacerbated for evaluating the edgeweights as displayed in Table 4. In Table 4, a “+” in the lower-triangle indicates the estimated edgeweight is significantly greater than chance, a “−” indicates the estimated edgeweight is less than chance, and a blank indicates that the value fell within the 99% confidence interval. One of the first things to notice is the prevalence of minus signs in Table 4. This indicates the absence of a link is just as important as the presence of a link. Of the 153 possible links, only 99 were significantly different than chance (65%), indicating that approximately one third of the estimated network is functioning equivalently to what we would expect by chance. Furthermore, with the three possible classifications of any individual edgeweight being (a) significantly greater than expected by chance, (b) significantly less than expected by chance, and (c) no different than expected by chance, the edgeweights are classified into those categories at the almost equal rates of 30%, 35%, and 35%, respectively. This uniform distribution of edgeweights to the three possible outcomes indicates that evaluating the results from these models by mere inspection of the edgeweights themselves is noninformative.

Estimates and Significance for Edge Weights

Once again, however, the specific edgeweights that are significant cannot be deduced by any pattern of edgeweights or their raw magnitude. For instance, the probability of an edgeweight being no different than chance if the observed edgeweight is greater than zero is 44%—so an observed edgeweight greater than zero is almost as equally as likely to be due to chance as not. To make it more salient, that would mean that nearly half of the edges depicted on the left side of Figure 3 in the Forbes et al. article are due to chance alone. Additionally, some of the zeros are included in the confidence intervals and some are not. Specifically, the probability of an edgeweight being significantly worse than chance given the observed edgeweight is less than or equal to zero is 28%. In terms of the graphical representation, that would mean that nearly 3 of 10 of the lines that are absent on the left hand side of Figure 3 are absent because of chance. Such variability in the confidence of the fidelity of observed edges in the figure renders these types of depictions of networks almost useless. This calls for an implementation of a formal testing procedure to assess the relevance of reported edgeweights or their absence.

Figure 2 illustrates the four possible outcomes as described in Table 1. The upper-left panel indicates an “interesting” finding between depressed mood and loss of interest. The edgeweight (1.93) falls above the distribution for random chance, and the bootstrap confidence interval around the edgeweight does not include zero. The lower-left panel indicates a “potentially interesting” connection between sleep problems (depression) and anxiety about more than one event. Specifically, the estimated value of zero is nonsignificant by both the eLasso and the bootstrap confidence interval around the estimate (0.00); however, the distribution based on random chance alone would expect this connection to be greater than what was observed. The final two panels on the right indicate two different “uninteresting” scenarios. The lower right panel shows the distribution for the edgeweight between chronic anxiety and sleep problems from anxiety—both distributions encompass the observed estimate of zero. The upper-right panel indicates a situation, in this case the relationship between loss of interest and psychomotor disturbances. The bootstrap test indicates that the edgeweight is significantly different than zero; however, the distribution derived from random matrices indicates that the observed value of 0.52 is not unexpected.

Figure 2. Examples of bootstrap compared with random distribution.

Confidence Levels

Above, we used a 99% confidence level to help control for multiple tests and illustrate the general notion of the differences between generating confidence intervals that reflect accuracy and stability through a bootstrap procedure versus the types of confidence intervals that aid in determining whether effects are different than would be expected by random chance alone. However, it is imperative to realize that even a more stringent confidence interval level will not aid in overcoming the threat of Type I error rates in the context of so many tests. Generally, there are different tests for edgeweights and 3p tests for centrality measures (e.g., one for each variable on strength, closeness, and betweenness). In Table 4, there are unique effects—for an α = .05, there would be (.05)153 = 7.65 significant effects based on chance alone, with the more stringent α = .01, (.01)(153) = 1.53 significant effects are expected to be attributable to chance alone.

In situations that rely on multiple testing, it is common to correct each individual test’s alpha to obtain an overall familywise α that corresponds to the prespecified value. The most conservative adjustment is the Bonferroni correction, which divides the familywise α by the total number of tests to obtain an appropriate level of α for each individual test—in this case, the adjusted α is (.05)/153 = .0003. As Epskamp, Borsboom, and Fried (in press) indicate, this type of correction can be overly burdensome and lead to the possibility of not finding any of the true effects (e.g., Type II errors). Beyond the inherent inverse relationship between Type I and Type II error, the computational burden to generate the requisite number of bootstrap samples for such small values of α can result in the procedure not being used in practice. For instance, generating enough samples to be able to obtain a 99.97% confidence interval would require around 10,000 samples; however, somewhere in the neighborhood of 50,000–10,000 would be preferable. As such, in their recommendations, Epskamp et al. (in press) indicate that corrections for chance will be relegated to future research, and in the interim, it is appropriate to proceed with uncorrected tests for the edgeweights.

While the sentiment to proceed with testing each test at α = .05 (the default setting in the Espkamp et al. bootstrapping software) is understandable, cautions, nonetheless, are called for. First, the probability of committing at least one Type I error when conducting multiple tests is 1 - (1 - α)^T, where T is the number of tests being conducted. For instance, if you conduct two tests and do not correct for α the probability of committing at least one Type I error is 1 − (.95)² = .0975; however, for the example above, the probability of making at least one Type I error is 1 − (.95)¹⁵³ = .9997—an almost certainty. Furthermore, it is well known that the magnitude of p values cannot be compared to determine which effects are more likely to be significant and which are more likely to be either insignificant or a potentially a Type I error. The inability to do such a ranking is readily seen if one considers alternative correction methods beyond the Bonferroni correction. Specifically, the Holm-Bonferroni method (Holm, 1979) is uniformly more powerful than the Bonferroni correction; however, it proceeds by comparing the lowest p value to the strictest criterion, and then, as p values increase, the comparison criteria becomes less stringent. A similar test that is more powerful, but require assumptions regarding the joint distribution of the test statistics is Hochberg’s step-up procedure (Hochberg, 1988); likewise, one could consider Hommel’s (1988) stagewise rejective multiple test procedure. Unfortunately, each of these procedures would require in increase in the number of bootstrap samples—which would lead to the potential relaxation of the requirement to control for the familywise error rate.

Beyond the potential to incorrectly assume that effects are significant when they are not, the inattention to appropriately control for multiple testing has the potential to call into question all prior research in an area. This can be seen best with the application of multiple testing in fMRI research, another area that relies on a large number of tests and constructs substantive theory from patterns of significance. Recently, Eklund, Nichols, and Knutsson (2016) demonstrated that somewhere in the neighborhood of 40,000 fMRI studies may be invalid due to the inability to control for Type I error rates appropriately. Furthermore, in their conclusion the authors make a particularly relevant statement: “Although we note that metaanalysis can play an important role in teasing apart false-positive findings from consistent results, that does not mitigate the need for accurate inferential tools that give valid results for each and every study.” Because we are in the early stages of developing psychological networks, the most prudent course of action is also establishing appropriate inferential tools immediately and prevent the need to potentially correct hundreds or even thousands of published studies 5, 10, or 20 years from now. This view of tightening the reins for publication is further supported by the recommendation of a group of statisticians who recently weighed in on the reproducibility of psychological science (Johnson et al., 2017): “The results of this reanalysis provide a compelling argument for both increasing the threshold required for declaring scientific discoveries and for adopting statistical summaries of evidence that account for the high proportion of tested hypotheses that are false.” In fact, they suspect that potentially 90% (!!!) of tests performed in psychology experiments are testing negligible effects.

Fidelity of Data

Lastly, we note that there are several decisions made at the data analytic level that can affect and alter the results of any given network model. Some examples would include how the data are collected or how the responses are coded. For instance, it is possible that specific criteria could be coded differently depending on the interview employed to collect the data. Another example would be the so-called “skip out” issue that can occur in epidemiological questionnaires. For “skip out” items, there are gateway items that are assessed first—if the response to the gateway item is negative, then the remainder of the items are not asked, whereas if the response to the gateway item is affirmative, then all subsequent related items are asked. A question arises on how to handle the subsequent items if they were not assessed. A common approach is to assume that if the gateway item is not endorsed, then all subsequent subitems are not endorsed either—this is akin to a logical data imputation. Whether this is appropriate or not will depend on the specific set of questions and criteria being assessed. If it is inappropriate, the estimated effects will be biased; however, that being said, these biases will not effect the validity of the testing approach described above, nor does it mitigate the issues with not controlling for multiple testing.

Replicability

From these findings, we see that the results of many network analyses, as conducted on binary data, might be overstating their findings. Although space limitations only allowed for a modest analysis of the performance the Isingfit model on one data set, current work is expanding the investigation to a broader class of binary matrices as well as extending to the three other types of network models mentioned by Forbes et al. (2017). Preliminary results indicate that the findings will be similar to what was observed for the Isingfit model. In short, we second the conclusion of Forbes et al. and offer the addendum that discovering believable results, at the specific symptom level (whether that is the relationship between pairs of symptoms) or variable level statistics (e.g., centrality statistics), will likely be much more difficult than previously envisioned. As shown above, this is because empirical findings are difficult to distinguish from random chance, and we do not believe that it would be too strong of a suggestion that previously published findings using this methodology should be reevaluated using the above testing procedure. Without this additional testing, future research based on existing findings will likely lead to a significant degree of nonreplicability as the findings are potentially no difference than chance. While the network approach represents an important alternative view of diagnostic systems that could provide new insights into both the basic structure of psychopathology and identify promising targets for intervention by virtue of their centrality, existing methods must be considered unproven. Research practitioners must appreciate the limitations of the existing state-of-the-art and develop and refine approaches likely to provide more robust and interpretable solutions.

Are Psychopathology “Networks” Actually Networks?

In the first footnote, we note that much of the terminology in recent psychopathology network analysis has been borrowed from the traditional network analysis literature—much of which is rooted in psychology and sociology (see Wasserman & Faust, 1994). To determine whether the transferability of methods in traditional network analysis to psychopathology networks is warranted (or should be taken at face value), it is worth highlighting the differences between the two. Generally, there are two types of networks that can be considered: (1) networks that directly assess the relationships between the same set of observations (e.g., one-mode matrices as described above), and (2) affiliation networks where the connections are assessed between two sets of observations (two-mode matrices as described above). Clearly, psychopathology networks fall into the class of affiliation matrices where the connections are measured between observation and diagnostic criteria. The relationships between the criteria are then then derived by transforming the two-mode affiliation matrix to a one-mode so-called “overlap/similarity” matrix between the criteria, where traditional network methods are applied to this overlap/similarity matrix. Faust (1997) indicated that potential pitfalls arise when applying standard centrality measures to networks derived from affiliation matrices: “In going from the affiliation relation to either the actor co-membership relation or the event overlap relation, one loses information about the patterns of affiliation between actors and events. Thus, one needs to be cautious when interpreting centralities for these one-mode relations” (p. 189). The concerns of Faust (1997) are related directly to the concerns raised in the introduction where the methods developed for one-mode networks are applied to the two-mode networks (e.g., affiliation network or bipartite graph).

As Johnson et al. (2017) argue, the editorial policies (and funding priorities) must be compelled to adapt to higher standards prior to putting their stamp of approval on results. Given the nature of problems that confront psychometric network modeling, including (but not limited to): (a) the possibility of observing “significant” effects that are not different than random chance, (b) the difficulties induced by conducting numerous significant tests, which is not controlled for via bootstrapping, and (c) the uncertainty regarding applying traditional methodology developed for one-mode networks to two-mode networks, we wholeheartedly agree with the recommendation by Forbes and company to turn a skeptical eye toward these models. Additionally, given the results from the example above and the theoretical issues that surround multiple testing and appropriate reference distributions, we do not agree with Epskamp, Borsboom, and Fried (in press) that it is reasonable to continue conducting tests at the α = .05 while we wait for methodologists to develop procedures that address the shortcomings. Rather, it is the other way around: wait until methodologists develop the appropriate fixes then proceed with fitting these models. Not following such an approach runs the very real risk of creating a series of publications that contain results that are not reproducible and likely no different than what is expected under one of the most basic models of chance in all of categorical data analysis.

Although we have provided numerous caveats and cautions to employing psychometric network models, we do believe that they are opening a potentially important area of research for us to consider. Specifically, the moving more to a causal structure model, as opposed to a common cause (such as latent variable) model holds great appeal as the parallelism to traditional medical models of diseases is enhanced. In our view, it is likely that the true model (or at least the best fitting models) will be a hybrid of network models and latent variable models. Further, although we provide extreme caution in using these methods to motivate theory, we do believe that such techniques can be a useful tool in the arsenal of the researcher when used to supplement a rich, substantive knowledge regarding the psychopathology being studied. Additionally, we note that by using correlations (or functions thereof) the network models have wed themselves closely to traditional latent variable models in terms of how the relationships between variables are conceptualized. However, there is a rich history in assessing the similarity between binary items in general and elements within a bipartite graph specifically. Broadening approaches beyond the logistic regression framework encapsulated by the Isingfit approach (and perhaps even abandoning the notion of correlation as the foundational measure of association) could open the entire field of psychometric networks to entirely new horizons. Finally, many of the concerns raised in this commentary could be alleviated if we move from the current state of exploratory psychometric networks to confirmatory psychometric networks, allowing a priori specified effects to be tested rather being hamstrung out of the gate by the need to correct for multiple testing.

¹ We note that there is a good deal of appropriation of terminology from the social network analysis literature used in these analyses; we would caution too much transference as the psychopathology networks are induced from bipartite graphs versus being observed directly.

² The eigenvalues from the completely random data are obtained by averaging over several—usually one thousand or more—sets of eigenvalues that have been obtained from independently generated random data.

³ Such an approach has been superceded by the derivation of a closed-form solution for comparing the equivalence of two partitions (Steinley, Brusco, & Hubert, 2016), but the basic logic still applies.

⁴ We note that most of the time v0 = 0.

⁵ Note that under many situations, the expected value of the effect is often zero (e.g., no effect). For instance, in the simple case of a t test or a linear regression model, generating data from random distributions—accomplished by randomly assigning group labels for the t test or randomly generating predictors uncorrelated with the dependent variable for regression—we would expect mean differences of zero and slopes of zero. The deviation from no effect for these psychological networks is due to the restriction of the range of the sample space of potential networks that is induced from the observed marginal distributions.

⁶ The between network importance algorithm would be implemented in a similar fashion, but because of space constraints we have not included an example here.

⁷ The data set was provided to us by the first author, Miriam Forbes, of the target article.

Admiraal, R., & Handcock, M. S. (2008). networksis: A package to simulate bipartite graphs with fixed marginals through sequential sampling. Journal of Statistical Software, 24, 1–21.

Anderson, B. S., Butts, C., & Carley, K. (1999). The interaction of size and density with graph-level indices. Social Networks, 21, 239–267.

Arabie, P., & Hubert, L. J. (1990). The bond energy algorithm revisited. IEEE Transactions on Systems, Man, and Cybernetics, 20, 268–274.

Arabie, P., Hubert, L., & Schleutermann, S. (1990). Blockmodels from the bond energy algorithm. Social Networks, 12, 99–126.

Baker, M. (2016). Is there a reproducibility crisis? A Nature survey lifts the lid on how researchers view the crisis rocking science and what they think will help. Nature, 533, 452–455.

Brusco, M. J., Shireman, E., & Steinley, D. (in press). A comparison of latent class, K-means, and K-median method for clustering dichotomous data. Psychological Methods.

Brusco, M. J., & Steinley, D. (2006). Inducing a blockmodel structure on two-mode data using seriation procedures. Journal of Mathematical Psychology, 50, 468–477.

Brusco, M. J., & Steinley, D. (2007a). Exact and approximate algorithms for part-machine clustering based on a relationship between interval graphs and Robinson matrices. IIE Transactions, 39, 925–935.

Brusco, M. J., & Steinley, D. (2007b). An evaluation of a variable-neighborhood search method for blockmodeling of two-mode binary matrices based on structural equivalence. Journal of Mathematical Psychology, 51, 325–338.

Brusco, M. J., & Steinley, D. (2009). Integer programs for one- and two-mode blockmodeling based on prespecified image matrices for structural and regular equivalence. Journal of Mathematical Psychology, 53, 577–585.

Brusco, M. J., & Steinley, D. (2011). A tabu search heuristic for deterministic two-mode blockmodeling of binary network matrices. Psychometrika, 76, 612–633.

Chen, Y., Diaconis, P., Holmes, S. P., & Liu, J. S. (2005). Sequential Monte Carlo methods for statistical analysis of tables. Journal of the American Statistical Association, 100, 109–120.

Connor, E. F., & Simberloff, D. (1979). The assembly of species communities: Chance or competition?Ecology, 60, 1132–1140.

Diamond, J. M. (1975). Assembly of species communities. In M.Cody & J.Diamond (Eds.), Ecology and evolution of communities (pp. 342–444). Cambridge, MA: Harvard University Press.

Eklund, A., Nichols, T. E., & Knutsson, H. (2016). Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proceedings of the National Academies of Science of the United States of America, 113, 7900–7905.

Epskamp, S., Borsboom, D., & Fried, E. I. (in press). Estimating psychological networks and their accuracy: A tutorial paper. Behavior Research Methods.

Faust, K. (1997). Centrality in affiliation networks. Social Networks, 19, 157–191.

Forbes, M. K., Wright, A. G. C., Markon, K. E., & Krueger, R. F. (2017). Evidence that psychopathology symptom networks have limited replicability. Journal of Abnormal Psychology, 126, 1011–1016.

Harrison, M. T., & Miller, J. W. (2013). Importance sampling for weighted binary random matrices with specified margins. arXiv:1301.3928 [stat.CO]

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800–802.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.

Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75, 383–386.

Horn, J. L. (1965). A rationale and technique for estimating the number of factors in factor analysis. Psychometrika, 30, 179–185.

Hunter, D. R., & Handcock, M. S. (2006). Inference in curved exponential family models for networks. Journal of Computational and Graphical Statistics, 3, 565–583.

Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2017). On the reproducibility of psychological science. Journal of the American Statistical Association, 112, 1–10.

Miller, J. W., & Harrison, M. T. (2013). Exact sampling and counting for fixed-margin matrices. The Annals of Statistics, 41, 1569–1592.

Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on the replicability in psychological science: A crisis of confidence?Perspectives on Psychological Science, 7, 528–530.

Rosenberg, S., Van Mechelen, I., & De Boeck, P. (1996). A hierarchical classes model: Theory and method with applications in psychology and psychopathology. In P.Arabie, L. J.Hubert, & G.De Soete (Eds.), Clustering and classification (pp. 123–155). River Edge, NJ: World Scientific.

Steinley, D. (2004). Properties of the Hubert-Arabie adjusted Rand index. Psychological Methods, 9, 386–396.

Steinley, D. (2006). Profiling local optima in K-means clustering: Developing a diagnostic technique. Psychological Methods, 11, 178–192.

Steinley, D. (2007). Validating clusters with the lower bound for sum of squares error. Psychometrika, 72, 93–106.

Steinley, D. (2008). Stability analysis in K-means clustering. British Journal of Mathematical and Statistical Psychology, 61, 255–273.

Steinley, D., & Brusco, M. J. (2011). Testing for validity and choosing the number of clusters in K-means clustering. Psychological Methods, 16, 285–297.

Steinley, D., Brusco, M. J., Hubert, L. (2016). The variance of the adjusted Rand index. Psychological Methods, 21, 261–272.

Wasserman, S. W., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge, UK: Cambridge University Press.

APPENDIX A: Algorithms for Testing Replicability when Marginal Distributions are Fixe

The ingredients for testing so-called “within” replicability are fairly straightforward. Namely, all that is required is the original data matrix Xn×p with n rows and p columns as well as the network statistic of interest, say θ̂. The goal is then to obtain the sampling distribution for θ under the null distribution that the location of the ones in the binary matrix X are random subject to the observed marginal distributions for both the rows and columns.

The algorithm proceeds as follows:

1. Compute θ̂ from X; compute r_x and c_x (the marginal distributions for the rows and columns, respectively, which in this case are just sums across the columns and rows).
2. Choose the number of random data matrices, R, to generate.
3. Generate the i^th random matrix, M_i, such that r_mi = r_x and c_Mi = c_x (e.g., the marginal distributions of the random matrices will be equivalent to those of the observed matrix, X).
4. Compute θ_i for i = 1, . . ., R. Order the θ’s from smallest to largest, letting the ordered vector of θ be denoted as θ^(o).
5. Create a confidence interval under the null distribution of random association as .
6. If the observed test statistic, θ̂, falls within the interval in Step 5 then we cannot reject the null. Consequently, we conclude the observed value could have arisen by random chance alone.

Algorithm for Between Replicability

The algorithm for testing so-called “between” replicability is almost identical. All that is required is a second data matrix, Yn2×p, with which we want to compare the original data matrix xn1×p. The process proceeds as follows:

1. Compute θ̂_XY from f(X, Y), where f(·) is a function of correspondence between the two structures uncovered from X and Y (for instance, the function could be the correlation of the edge weights between the two network structures). Additionally, compute r_x and c_x from X; compute from Y, r_Y and c_Y from Y.
2. Choose the number of pairs of random data matrices, R, to generate.
3. Generate the i^th pair of random matrices (M_i, N_i), such that r_Mi = r_x and c_Mi = c_x and r_Ni = r_Y and c_Ni = c_Y.
4. Compute θ_i for i = 1, . . ., R. Order the θ’s from smallest to largest, letting the ordered vector of θ be denoted as θ^(o).
5. Create a confidence interval under the null distribution of random association as .
6. If the observed test statistic, θ̂, falls within the interval in Step 5 then we cannot reject the null. Consequently, we conclude the observed value could have arisen by random chance alone.

Submitted: May 17, 2017 Revised: July 19, 2017 Accepted: July 20, 2017

A New Decisional Balance Measure of Motivation to Change Among At-Risk College Drinkers

Acknowledgement: Susan E. Collins’s time was supported by National Institute on Alcohol Abuse and Alcoholism (NIAAA) Institutional Training Grant T32AA007455 awarded to Mary E. Larimer at the University of Washington. This study was supported in part by NIAAA Grant AA12518 awarded to Kate B. Carey.

A big thanks goes to Dan J. Neal for his statistical insights and advice, to Carrie Luteran for assembling the data sets, and Sonia Kaur for her help with data entry. Thanks also to Sandra Eck for the helpful ongoing discussions about decisional balance measurement in new participant populations.

Decision making often involves consideration of a set of possible behavioral options and an evaluation of the consequences of each option. These consequences may range from desirable to undesirable effects, also referred to as pros and cons (Janis & Mann, 1977). A decisional balance has thus been operationalized as a representation of the pros and cons of a certain behavior and its potential alternatives.

The explicit consideration of the decisional balance—either as a written or counselor-facilitated exercise—was originally proposed to reduce decision-making errors by making people more cognizant of the decision-making process and the factors contributing to their decisions (Janis, 1968). In this context, decisional balance was designed as a therapeutic exercise to facilitate the complete and realistic assessment of the net value of a potential behavior. As this technique has evolved over time, clinicians have started to employ guided decisional balance exercises as a means of enhancing motivation to change risky health behaviors (Dimeff, Baer, Kivlahan, & Marlatt, 1999; Miller, 1999). Specifically, clients articulate and examine ambivalence about their current behavior to determine whether the weight of the evidence is accumulating toward the need for behavior change (Miller, 1999). In the college drinking literature, a limited number of studies have used a guided, open-ended decisional balance as an intervention tool, but the results of guided interventions have been mixed with regard to alcohol use outcomes (Carey, Carey, Maisto, & Henson, 2006; S. E. Collins & Carey, 2005; LaBrie, Pedersen, Earlywine, & Olsen, 2006).

Alternatively, the data generated in a decisional balance could reflect resolve to enter into a course of action (Janis & Mann, 1977), thus serving as a proxy for motivation to change. This idea was echoed in the work of DiClemente and colleagues (e.g., DiClemente et al., 1991), who have since recognized decisional balance as a marker for the initiation of different stages of change (Pollak, Carbonari, DiClemente, Niemann, & Mullen, 1998). In fact, the stages of change have been operationalized algorithmically as a function of change in the decisional balance (Hall & Rossi, 2008; Prochaska, 1994). Thus, the historical development of the decisional balance construct shows its potential to both enhance and reflect people’s motivational states. In this study, we focused on the role of decisional balance as an assessment tool rather than as an intervention procedure.

The existence of a relationship between decisional balance and motivation to change is evident; however, the nature of this relationship has been described differently in different theoretical contexts. Furthermore, how decisional balance is measured has included the consideration of the pros and cons of a behavior or the pros and cons of an alternative behavior. What exactly decisional balance measures, however, has been less clearly defined. For example, some researchers have asserted that it represents the decision-making process itself (Fischoff & Quadrel, 1991; Janis & Mann, 1977), others have posited that it represents one dimension of motivation to change (Miller, 1999), and yet others have suggested that it is a covariate or perhaps a mechanism involved in transitioning through various stages of behavior change (Prochaska et al., 1994). Considering the ambiguity in the relationship between motivation and decisional balance, more research is needed to establish the validity and clinical utility of decisional balance as a measure of motivation to change.

Reflecting the ambiguity surrounding the decisional balance construct, several decisional balance measures for drinking have been designed over the past 2 decades. The Alcohol Decisional Balance Scale assesses the pros and cons of maintaining one’s current alcohol use using a 42-item Likert scale questionnaire (King & DiClemente, 1993). The Alcohol and Drug Consequences Questionnaire is a 28-item, 6-point Likert scale questionnaire designed to assess the pros and cons of changing alcohol or drug behavior (Cunningham, Gavin, Sobell, Sobell, & Breslin, 1997). Finally, the Decisional Balance for Immoderate Drinking (DBID) was developed for college students (Migneault, Velicer, Prochaska, & Stevenson, 1999). This measure consists of 25 Likert scale questions assessing on a 5-point scale the importance of selected pros and cons of “immoderate” drinking. Different sets of these original 25 items may be summed to form either two 20-item scales (pros vs. cons of immoderate drinking) or three 18-item scales (pros, potential cons, and actual cons).

Despite initially promising psychometric evaluations of these questionnaires, conceptual weaknesses can be identified. First, all of these measures have focused exclusively on the pros and cons of current drinking or the pros and cons of reducing or changing drinking, which precludes the evaluation of the decisional balance as a whole. An incomplete decisional balance has been viewed as problematic in decision-making theory because of the potential for overlooked consequences to create new ambivalence after a decision has been reached (Janis & Mann, 1977). Similarly, an incomplete measure of decisional balance may fail to take into account all aspects of a person’s current motivation to change. One empirical study found that consideration of both the target behavior and an alternative behavior nearly doubled the number of pros and cons spontaneously produced across multiple risky behaviors (Beyth-Marom, Austin, Fischhoff, Palmgren, & Jacobs-Quadrel, 1992). Particularly among adolescents in that study, negative social consequences were reported more often in assessing the cons of not drinking. Thus, a focus on only one half of the decision-making process (i.e., either the pros and cons of current or alternative behavior) might lead to an incomplete and potentially less predictive measure of decisional balance.

Another problem with decisional balance measures to date is that the pros and cons are generated by researchers instead of by participants themselves. This approach may be inadequate in capturing motivation to change authentically and accurately. First, if researchers approach the topic from an academic perspective, they may identify different pros and cons of current drinking versus drinking behavior change than participants (Fischoff & Quadrel, 1991). Also, the language used by researchers to describe the pros and cons can, in and of itself, influences participants’ interpretation of the item and thus the participants’ answers (Beyth-Marom et al., 1992; Fischoff & Quadrel, 1991). On a related point, in providing participants with the “correct” pros and cons of drinking, researchers may be artificially constructing the decision-making process to which participants passively respond. This approach may have the unwanted side effect of making respondents aware of pros and cons they may not have otherwise considered and that may not represent their own unique decision-making process. In contrast, use of an open-ended response format allows participants to express their actual motivational state rather than respond to researchers’ perspectives and values (Fischoff & Quadrel, 1991).

In light of these concerns, it is plausible that an open-ended, participant-generated decisional balance could provide a more accurate measure of motivation to change and better predict drinking outcomes among college drinkers. In an exploratory analysis, Collins and Carey (2005) presented some evidence that, when used as an assessment of motivation, the pattern of responses generated during a decisional balance exercise predicted drinking outcomes with modest success. Specifically, among participants receiving an in-person decisional balance exercise, a greater proportion of pros to cons of changing one’s drinking predicted short-term drinking outcomes.

The current study was designed to extend these preliminary findings and examine an expanded, open-ended, four-field decisional balance worksheet as a measure of motivation to change drinking among at-risk college drinkers. The decisional balance worksheet prompted respondents to report the pros and cons of their current drinking versus reduced drinking. Next, the number of pros and cons reported in each field was converted into a proportion representing the decisional balance toward change or the decisional balance proportion (DBP).

The goal of this study was to test the validity of the DBP as a new measure of motivation to change drinking. We hypothesized that the DBP would evince convergent validity by positively correlating with an alternative, Likert scale measure of readiness to change, and positively correlating with the cons and negatively correlating with the pros of drinking as measured by continuous decisional balance scales. Furthermore, we hypothesized that the DBP would evince discriminant validity by showing nonsignificant correlations with measures assessing dissimilar constructs (i.e., social desirability and demographic variables). Finally, we hypothesized that DBP change scores would evince predictive validity. Initial increases in the decisional balance reflecting movement toward drinking behavior change would predict greater decreases in heavy drinking indices over the follow-up period compared with a DBP reflecting no movement or movement away from change.

Participants

Participants consisted of 143 undergraduate volunteers who had participated in a randomized clinical trial of two types of brief motivational interventions (see Carey et al., 2006). Inclusion criteria for this trial were (a) reporting at least one heavy drinking episode in an average week or at least four heavy drinking episodes in the past month, (b) being 18–25 years of age, (c) being a freshman, sophomore, or junior in college, and (d) consenting to participate. Only students who had participated in the first year of the larger study were included in the current secondary analyses because the format of the original measure was changed after the first year. The new format allowed for electronic data scanning but, as a consequence, limited the number of potential entries. Because the number of entries in each field of the decisional balance was the primary focus of the current study, the data set includes only the participants who had the opportunity to respond to the unrestricted format.

The subsample of 143 students in the current study was predominately female (68%, n = 97), and the average age was 19.20 years (SD = 0.87). The sample consisted of 65% freshmen, 27% sophomores, and 7% juniors. The majority self-identified as White (87%), whereas 2% self-identified as Black/African American, 5% as Asian/Pacific Islander, 3% as Hispanic/Latino/a, and 4% as other or multiracial. Membership in a fraternity or sorority was reported by 24% of the sample, and most participants reported living on campus (89%) or in a fraternity or sorority house (2%), as opposed to off campus (8%).

Measures

A set of demographic questions assessed participants’ age, gender, year in college, ethnicity, on- or off-campus residence, and membership in an on-campus Greek organization. Social desirability was measured using a 13-item short form of the Marlowe Crowne Social Desirability Scale (Reynolds, 1982). The alpha in the current sample was adequate (α = .63).

The decisional balance worksheet was modeled after a scale used to assess the accessibility of alcohol expectancies (see Stacy, Leigh, & Weingardt, 1994). Because it is a free-recall task, it was administered prior to other Likert scale measures of decisional balance. Participants recorded each “advantage” and “disadvantage” of “continuing to drink as you are now” and “drinking less than you do now” on prenumbered lines on the open-ended decisional balance worksheet. The counts of the pros and cons were obtained by summing the prenumbered lines filled in by participants and formed the main explanatory variable for this study, DBP, which may be written as   where subscripts red = reducing drinking and cur = current drinking. DBP scores at 0.5 represent an even balance between pros and cons of reducing drinking and current drinking. Scores between 0.5 and 1.0 indicate a balance tipped toward reducing drinking, and DBP scores between 0.0 and 0.5 indicate a balance tipped toward maintenance of current drinking. In the current study, baseline DBP was used in the convergent and discriminant validity tests, whereas change in the DBP from baseline to 1-month follow-up was used in predictive validity tests.

The 12-item Readiness to Change Questionnaire (RTCQ; Rollnick, Heather, Gold, & Hall, 1992) was scored as a continuous measure of readiness to change (Budd & Rollnick, 1996). The RTCQ was used in convergent and predictive validity analyses in the current study. The alpha reached an adequate level of item consistency (α = .80).

The DBID (Migneault et al., 1999) is a 25-item Likert scale measure developed for college students. Different sets of these original 25 items may be summed to form either two scales (10-item Pros vs. 10-item Cons of immoderate drinking) or three scales (10-item Pros, 3-item Potential Cons, and 5-item Actual Cons). The 18-item, three-factor solution was used in tests of convergent and predictive validity in the current study. In the overall sample, reliability was calculated for Pros (α = .81), Potential Cons (α = .71), and Actual Cons (α = .52).

All drinking assessments used the previous 30 days as a uniform time frame and defined a drink as a 10- to 12-oz can or bottle of 4%–5% beer, a 4-oz glass of 12% table wine, a 12-oz bottle or can of wine cooler, or a 1.25-oz shot of 80-proof liquor either straight or in a mixed drink. A modified version of the Daily Drinking Questionnaire (R. L. Collins, Parks, & Marlatt, 1985) allowed for calculation of drinking frequency and quantity per heaviest drinking week. Using this measure, participants also estimated the frequency of heavy episodic drinking (HED), defined as five or more drinks for men and four or more drinks for women on one occasion (Wechsler et al., 2002). This measure yielded three of the alcohol outcome variables.

The Rutgers Alcohol Problem Index (RAPI; White & Labouvie, 1989) consists of 23 items assessing alcohol-related problems and was specifically developed for use with adolescents and young adults. Participants used a Likert scale to indicate how many times in the past 30 days they experienced each problem listed (i.e., 0 = 0 times, 1 = 1–2 times, 2 = 3–5 times, 3 = 6–10 times, 4 = more than 10 times), and a summary score represented severity of problems. Adequate internal consistency was obtained in the overall sample (α = .82).

Procedure

Participants attending group baseline sessions provided informed consent and completed the measures mentioned above; all received course credit for their participation. Those who reported engaging in HED at least 4 times in the past month were invited via telephone to participate in a randomized clinical trial that included various brief intervention and assessment conditions (for details, see Carey et al., 2006). Following the intervention period, participants were invited to attend 1-, 6-, and 12-month in-person follow-up assessments for which they filled out the same questionnaires and received course credit, $20, and $25, respectively.

Preliminary Data Analysis

The distributions of the explanatory and outcome variables were examined for univariate outliers and deviation from the expected distributions. All drinking outcome measures were positively skewed count variables that approximated the negative binomial distribution. There were no extreme univariate outliers (see Table 1 for all means and standard deviations).

Descriptive Statistics for Explanatory and Response Variables (N = 143)

Convergent Validity

To determine the convergent validity of the DBP, we conducted bivariate Spearman correlations between the baseline DBP and baseline summary scores for the RTCQ, DBID–Pros, DBID–Potential Cons, and DBID–Actual Cons. Although the correlations with the DBP were all in the expected directions, only the correlations between the DBP and RTCQ and DBID–Actual cons were statistically significant (see Table 2 for correlations).

Bivariate Correlations Between the DBP, RTCQ, and DBID Scales

Discriminant Validity

Bivariate Spearman correlations and Mann–Whitney U tests were conducted to test the hypothesized lack of association between the baseline DBP and gender, U(N = 143) = –0.26, p = .80; race/ethnicity, U(N = 143) = –1.32, p = .19; housing situation, U(N = 142) = 0.78, p = .44; social desirability, ρ = –0.09, p = .28; age, ρ = –0.12, p = .14; and Greek membership, U(N = 143) = –1.03, p = .30. As all tests were nonsignificant, discriminant validity of this measure was supported.

Predictive Validity of Change in Decisional Balance

Analysis plan

Population-averaged generalized estimating equation (Zeger & Liang, 1986) models were conducted using STATA 10 (StataCorp, 2007) and tested the change in DBP from baseline to 1-month follow-up as a predictor of heavy drinking outcomes over the 12-month follow-up period. For those unfamiliar with population-averaged generalized estimating equation models, they may be conceptualized as marginal regression models that can be applied to data conforming to different types of distributions (e.g., normal, Poisson, negative binomial, binomial) and can take into account nonindependence resulting from data clustering (e.g., longitudinal data collected on one participant).

The outcome variables were based on a time frame of the previous 30 days and included drinking quantity and frequency during the heaviest drinking week, HED frequency, and RAPI score. Because the distributions of the drinking outcome variables were positively skewed, and the variables were typically overdispersed count/integer responses (e.g., number of drinks consumed during the heaviest drinking week), negative binomial distributions were specified (cf. Neal & Simons, 2007). To enhance interpretability of the regression coefficients, we used the log link for these models. For all variables, repeated measures on one case served as the clustering variable. Because the drinking outcome variables were longitudinal, unevenly spaced, and variably correlated, we used an unspecified correlation structure (Hardin & Hilbe, 2003). The unspecified correlation structure allowed for the correlation in the drinking outcome data at each time point to be taken into account in the overall model estimation.

Three separate models for each of the drinking outcome variables were used to test the relative predictive abilities of DBP, RTCQ, and DBID change scores. The DBP models included five predictors: a linear time variable that compared drinking outcomes at baseline, 1-, 6-, and 12-month follow-ups (coded as 0, 1, 6, and 12, respectively); a quadratic time variable, which took into account the fact that alcohol use over time often follows a curvilinear versus a straight linear path; the DBP change score, which reflected movement in the balance toward or away from change (1-month follow-up minus baseline DBP); and both Linear and Quadratic Time × DBP interactions. Similar but nonnested models involving change on the RTCQ scale and on the three DBID scales were also run and were subsequently compared on goodness of fit using the quasi-likelihood under the independence model information criterion (QICu; Hardin & Hilbe, 2003). Similar to the Akaike’s information criterion tests, statistically superior models have the lowest QICu scores (Hardin & Hilbe, 2003).

Quantity: Heaviest drinking week

The DBP model (QICu = 5124.78) for quantity during the heaviest drinking week was significant, Wald χ²(5, N = 143) = 14.16, p = .01, and statistically superior to both the RTCQ (QICu = 14458.37) and DBID (QICu = 17711.27) models. After controlling for time and baseline DBP, there were significant Linear Time × DPB (IRR = 0.78, SE = 0.07, p = .006) and Quadratic Time × DBP (IRR = 1.02, SE = 0.007, p = .007) interactions. As shown in Figure 1, all participants seemed to decrease their heavy drinking quantity between baseline and 1-month follow-up, possibly as a result of assessment reactivity. However, increases in DBP scores over the initial 1-month period predicted subsequent decreases in drinking quantity at the 6-month follow-up, an effect that decayed by the 12-month follow-up. Decreasing DBP scores tended to predict an increase in drinking quantity, which was followed by a reduction at the 12-month follow-up. On the other hand, relatively stable DBPs were associated with little change in drinking quantity.

Figure 1. Graph of mean quantity per heaviest drinking week by time point and level of change in decisional balance proportion (DBP). The DBP change scores represent change in the DBP between baseline and the 1-month follow-up. For clarity of presentation, groups were formed to represent different levels of change in the DBP. The stable DBP group in the figure is centered on the mean DBP change score (M = –0.004, SD = 0.20) in this sample and includes difference scores ranging from –0.20 to 0.20. These scores correspond to 1 SD below and 1 SD above the mean, respectively. The decreasing DBP group represents participants whose DBP change scores were at least 1 SD below the mean (DBP < –0.20), and the increasing DBP group represents participants whose DBP change scores were at least 1 SD above the mean (DBP > 0.20).

Frequency: Heaviest drinking week

The DBP model (QICu = 343.23) for frequency during the heaviest drinking week was statistically superior to both the RTCQ (QICu = 1100.58) and DBID (QICu = 1280.73) models, and was significant, Wald χ²(5, N = 143) = 15.36, p = .009. After controlling for time and baseline DBP, there were significant Linear Time × DPB (IRR = 0.84, SE = 0.05, p = .003) and Quadratic Time × DBP (IRR = 1.01, SE = 0.005, p = .005) interactions. After all groups initially decreased on heavy drinking frequency, increasing DBP over the initial 1-month period predicted greater decreases in drinking frequency during the heaviest drinking week—until the 12-month follow-up, when this decreasing effect decayed (see Figure 2). Decreasing DBP scores tended to predict increases in drinking frequency up to the 6-month time point, followed by a reduction at the 12-month follow-up. On the other hand, relatively stable DBPs were associated with little change in drinking frequency.

Figure 2. Graph of mean frequency per heaviest drinking week by time point and level of change in decisional balance proportion (DBP).

HED

The best model according to the QICu was the DPB model (QICu = 1481.03) compared with the RTCQ (QICu = 4069.71) and DBID (QICu = 4891.80) models; however, none of the omnibus model tests for HED outcomes were significant (all ps > .18).

RAPI

Compared with the RTCQ (QICu = 5375.68) and DBID (QICu = 6493.62) models, the DBP (QICu = 1975.94) model provided the best fit for self-reported alcohol-related problems, Wald χ²(5, N = 143) = 13.12, p = .02. After controlling for time and baseline DBP, there was a significant Linear Time × DPB interaction (IRR = 0.85, SE= 0.07, p = .046). As shown in Figure 3, all participants reported initial decreases in RAPI. Increases in DBP over the initial 1-month period, however, predicted a stable linear decrease in alcohol-related problems over the follow-up period. On the other hand, decreasing DBP scores tended to predict increases in alcohol-related problems up to the 6-month time point, followed by a downward trend at the 12-month follow-up. On the other hand, relatively stable DBPs are associated with relatively stable experience of alcohol-related problems.

Figure 3. Graph of mean alcohol-related problems (Rutgers Alcohol Problem Index [RAPI] score) by time point and level of change in decisional balance proportion (DBP).

This study provided an examination of a DBP as a measure of motivation to change among at-risk college drinkers. The DBP was generated from responses to an open-ended decisional balance worksheet assessing pros and cons of current drinking versus reducing drinking; it was constructed to reflect the extent to which the decisional balance was tipped toward change.

Convergent validity of the DBP was partially supported in this study. As predicted, initial DBP positively and significantly correlated with readiness to change as measured by the RTCQ. This finding provided convergent validity for the DBP as a measure of motivation to change. Furthermore, the weighted importance of current negative outcomes (DBID–Actual Cons) in participants’ decisions to drink was significantly, albeit weakly, associated with DBP scores. The somewhat weak effect may indicate that the DBID and the DBP measure overlapping yet distinct constructs, or it may reflect the relatively low reliability of the DBID–Actual Cons scale. The latter point is a psychometric issue that may have limited the power to optimally assess convergent validity with this scale.

The DBP was not significantly correlated with the pros scale of the DBID, which taps into the importance of positive aspects of “immoderate” drinking (e.g., “I feel happier when I drink”). However, this nonsignificant correlation is understandable: The DBP was constructed to reflect the tilt of the decisional balance toward change, not the status quo. In fact, the DBP represents the weight of the cons of current drinking plus the pros of changing relative to all fields in the balance. Because this proportion does not explicitly highlight the pros of current drinking, they may be “passively” outweighed.

The fact that the DBP was not correlated with the DBID–Potential Cons scale reflects the mixed findings regarding convergent validity of the DBP. However, this lack of association may also be interpreted in the context of the cognitive and memory literature, which asserts that individual drinking experience influences the accessibility of certain thoughts about alcohol use. Specifically, frequently encountered outcomes, such as those represented by the DBID–Actual Cons scale (e.g., “Drinking makes me feel out of control”), may be more accessible than hypothetical, potential outcomes, such as those measured by the DBID–Potential Cons scale (e.g., “Drinking could kill me”; Stacy et al., 1994). On the other hand, the DBP, which reflects an open-ended assessment of the decisional balance, may be even more accurate in assessing the most personally salient pros and cons of a behavior and may be more reflective of an individual’s current motivational state than the DBID scales.

Evidence for the discriminant validity of the DBP was obtained. Our findings confirmed that the DBP did not significantly correlate with demographic measures (i.e., gender, race/ethnicity, year in college, Greek membership, age) or social desirability. Perusal of the recent literature reveals no studies involving college students that have demonstrated associations between motivation and basic demographic variables or social desirability. Thus, consistent with empirical precedent and theory, the DBP appears to be an independent construct that may be used with a range of college students.

Findings in this study revealed that changes in the DBP predicted heavy drinking outcomes among at-risk college drinkers. In fact, the DBP models fit the data better than did models including RTCQ and DBID change scores as predictors of alcohol use over time. Furthermore, the DBP models were the only ones that yielded consistently significant predictors of drinking outcomes. Across three of the four drinking outcome variables, movement toward change (i.e., an increase in proportion of pros of reduced drinking and cons of current drinking to total item count from baseline to 1-month follow-up) predicted reductions in drinking over the initial follow-up period.

However, this finding was tempered by a curvilinear effect on two of the four drinking outcome variables, drinking quantity and frequency, which complicated the initial linear findings. After participants with increasing motivation to change initially decreased their heavy drinking quantity and frequency (and vice versa), there was an apparent decay in this effect at the 12-month follow-up. This regression to the mean, however, does not necessarily indicate that the DBP does not predict drinking as hypothesized. It is highly possible that the relatively short-term changes in the DBP that occurred initially from baseline to 1-month posttest may be most helpful in predicting more proximal changes in drinking. Thus, the more distal drinking measured 1 year after the initial motivation ratings may not be as reliably predicted by the DBP. This explanation fits with the literature on motivation, which suggests that motivation to change drinking is a fluid state rather than a stable trait (Miller, 1999). It also corresponds to the developmental literature that documents temporal variation in college student drinking patterns over weeks and years (Del Boca, Darkes, Greenbaum, & Goldman, 2004; Schulenberg, O’Malley, Bachman, Wadsworth, & Johnston, 1996). Further studies are needed to establish the temporal robustness of changes in motivation as measured by the DBP. Perhaps models assessing the time-varying and parallel change in DBP and drinking would be a helpful next step.

Conceptually, the DBP corresponds to decisional balance and motivation to change theory better than the other measures (i.e., RTCQ and DBID) tested as predictors in this study. Unlike previous studies involving decisional balance measures (e.g., Cunningham et al., 1997; King & DiClemente, 1993; Migneault et al., 1999; Velicer, DiClemente, Prochaska, & Brandenburg, 1985), the DBP integrates all four fields of the decisional balance: the pros and cons of both current drinking and drinking reduction. Because both theory and empirical findings have indicated that the consideration of the pros and cons of both current behavior as well as behavior change are key to accurate assessment of a person’s current motivation to change (Beyth-Marom et al., 1992; Janis & Mann, 1977), the use of the decisional balance worksheet may represent a step forward in decisional balance measurement. Furthermore, the fact that the input for the DBP is participant- instead of researcher-generated may make this measure a more accurate and personally relevant representation of one’s motivation to change than the RTCQ and DBID (Fischoff & Quadrel, 1991). Finally, the open-ended format of the decisional balance worksheet lends itself to potential qualitative as well as quantitative representations of motivation to change.

Limitations

This study comprised a nonrandom sample of at-risk college drinkers who had participated in a larger intervention trial. Considering the potential confounding effects of the nonrandom selection and exposure to brief interventions, it is necessary to replicate these results on a larger, randomly selected, nontreatment sample. Furthermore, the relatively homogeneous racial and ethnic composition of the current sample raises questions as to the external validity of the current findings. This sample consisted of predominantly White, non-Hispanic students; thus, further study of the DBP and its ability to predict drinking outcomes in more diverse samples is necessary to ensure its generalizability to other populations. We also recognize that the DBP focuses on numbers of items generated rather than their content. Although potential information may be gained by considering item content as well, the DBP has the advantage of rapid and reliable scoring. Despite these limitations, the current results provide additional support for and expansion on a quantification of a drinking decisional balance originally introduced by Collins and Carey (2005).

Conclusions

This study has provided evidence for the convergent, discriminant, and predictive validity of a new decisional balance measure of motivation to change drinking behavior among at-risk college drinkers. This study adds to the literature because previous conceptualizations of the decisional balance measure were researcher- instead of participant-generated and were limited in their scope (i.e., assessed either pros and cons of drinking or of changing behavior but not both). Furthermore, this measure appears to predict longitudinal drinking better than established, Likert scale measures of readiness to change and decisional balance. Larger scale studies should be conducted to provide additional support for the psychometric integrity, clinical utility, and generalizability of this decisional balance measure.

Beyth-Marom, R., Austin, L., Fischhoff, B., Palmgren, C., & Jacobs-Quadrel, M. (1992). Perceived consequences of risky behaviors: Adults and adolescents. Developmental Psychology, 29, 549–563.

Budd, R. J., & Rollnick, S. (1996). The structure of the Readiness to Change Questionnaire: A test of Prochaska & DiClemente’s transtheoretical model. British Journal of Health Psychology, 1, 365–376.

Carey, K. B., Carey, M. P., Maisto, S. A., & Henson, J. M. (2006). Brief motivational interventions for heavy college drinkers: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 74, 943–954.

Collins, R. L., Parks, G. A., & Marlatt, G. A. (1985). Social determinants of alcohol consumption: The effects of social interaction and model status on the self administration of alcohol. Journal of Consulting and Clinical Psychology, 53, 189–200.

Collins, S. E., & Carey, K. B. (2005). Lack of effect for decisional balance as a brief motivational intervention for at-risk college drinkers. Addictive Behaviors, 30, 1425–1430.

Cunningham, J. A., Gavin, D., Sobell, L. C., Sobell, M. B., & Breslin, F. C. (1997). Assessing motivation for change: Preliminary development and evaluation of a scale measuring the costs and benefits of changing alcohol or drug use. Psychology of Addictive Behaviors, 11, 107–114.

Del Boca, F. K., Darkes, J., Greenbaum, P. E., & Goldman, M. S. (2004). Up close and personal: Temporal variability in the drinking of individual college students during their first year. Journal of Consulting and Clinical Psychology, 72, 155–164.

DiClemente, C. C., Prochaska, J. O., Fairhurst, S. K., Velicer, W. F., Velasquez, M. M., & Rossi, J. S. (1991). The process of smoking cessation: An analysis of precontemplation, contemplation, and preparation stages of change. Journal of Consulting and Clinical Psychology, 59, 295–304.

Dimeff, L. A., Baer, J. S., Kivlahan, D. R., & Marlatt, G. A. (1999). Brief alcohol screening and intervention for college students (BASICS): A harm reduction approach. New York: Guilford Press.

Fischoff, B., & Quadrel, M. J. (1991). Adolescent alcohol decisions. Alcohol Health and Research World, 15, 43–51.

Hall, K. L., & Rossi, J. S. (2008). Meta-analytic examination of the strong and weak principles across 48 health behaviors. Preventive Medicine, 46, 266–274.

Hardin, J. W., & Hilbe, J. M. (2003). Generalized estimating equations. Boca Raton, FL: Chapman & Hall/CRC.

Janis, I. L. (1968). Pilot studies on new procedures for improving the quality of decision making. New Haven, CT: Yale Studies in Attitudes and Decisions.

Janis, I. L., & Mann, L. (1977). Decision-making: A psychological analysis of conflict, choice, and commitment. New York: The Free Press.

King, T. K., & DiClemente, C. C. (1993, November). A decisional balance measure for assessing and predicting drinking behavior. Paper presented at the Annual Conference of the Association for the Advancement of Behavior Therapy, Atlanta, GA.

LaBrie, J. W., Pedersen, E. R., Earlywine, M., & Olsen, H. (2006). Reducing heavy drinking in college males with the decisional balance: Analyzing an element of motivational interviewing. Addictive Behaviors, 31, 254–263.

Migneault, J. P., Velicer, W. F., Prochaska, J. O., & Stevenson, J. F. (1999). Decisional balance for immoderate drinking in college students. Substance Use & Misuse, 34, 1325–1346.

Miller, W. R. (1999). Enhancing motivation for change in substance abuse treatment (Vol. 35). Rockville, MD: U.S. Department of Health and Human Services.

Neal, D. J., & Simons, J. S. (2007). Inference in regression models of heavily skewed alcohol use data: A comparison of ordinary least squares, generalized linear models, and bootstrap resampling. Psychology of Addictive Behavior, 21, 441–452.

Pollak, K. I., Carbonari, J. P., DiClemente, C. C., Niemann, Y. F., & Mullen, P. D. (1998). Causal relationships of processes of change and decisional balance: Stage-specific models for smoking. Addictive Behaviors, 23, 437–448.

Prochaska, J. O. (1994). Strong and weak principles for progressing from precontemplation to action on the basis of twelve problem behaviors. Health Psychology, 13, 47–51.

Prochaska, J. O., Velicer, W. F., Rossi, J. S., Goldstein, M. G., Marcus, B. H., Rakowski, W., et al. (1994). Stages of change and decisional balance for 12 problem behaviors. Health Psychology, 13, 39–46.

Reynolds, W. M. (1982). Development of reliable and valid short forms of the Marlowe–Crowne Scale of Social Desirability. Journal of Clinical Psychology, 38, 119–125.

Rollnick, S., Heather, N., Gold, R., & Hall, W. (1992). Development of a short “Readiness to Change Questionnaire” for use in brief, opportunistic interventions among excessive drinkers. British Journal of Addictions, 87, 743–754.

Schulenberg, J., O’Malley, P. M., Bachman, J. G., Wadsworth, K. N., & Johnston, L. D. (1996). Getting drunk and growing up: Trajectories of frequent binge drinking during the transition to young adulthood. Journal of Studies on Alcohol, 57, 289–304.

Stacy, A., Leigh, B. C., & Weingardt, K. R. (1994). Memory accessibility and association of alcohol use and its positive outcomes. Experimental and Clinical Psychopharmacology, 2, 269–282.

StataCorp. Stata statistical software (Release 10) [computer software]. College Station, TX: StataCorp.

Velicer, W. F., DiClemente, C. C., Prochaska, J. O., & Brandenburg, N. (1985). Decisional balance measure for assessing and predicting smoking status. Journal of Personality and Social Psychology, 48, 1279–1289.

Wechsler, H., Lee, J. E., Kuo, M., Seibring, M., Nelson, T. F., & Lee, H. (2002). Trends in college binge drinking during a period of increased prevention efforts. Journal of American College Health, 50, 203–217.

White, H. R., & Labouvie, E. W. (1989). Towards the assessment of adolescent problem drinking. Journal of Studies on Alcohol, 50, 30–37.

Zeger, S. L., & Liang, K.-Y. (1986). Longitudinal data analysis for discrete and continuous outcomes. Biometrics, 42, 121–130.

Submitted: October 1, 2008 Revised: January 1, 2009 Accepted: February 20, 2009

A Prospective Study of Pediatric Loss of Control Eating and Psychological Outcomes

Acknowledgement: Jack A. Yanovski is a commissioned officer in the U.S. Public Health Service (PHS). The opinions and assertions expressed herein are those of the authors and are not to be construed as reflecting the views of the PHS, the Uniformed Services University of the Health Sciences (USUHS), or the U.S. Department of Defense. The authors report no competing interests. Research support was provided by National Institutes of Health Intramural Research Program Grant Z01-HD-00641 from the Eunice Kennedy Shriver National Institute of Child Health and Human Development to Jack A. Yanovski, supplemental funding from the National Center on Minority Health and Health Disparities to Jack A. Yanovski, National Institute of Diabetes and Digestive and Kidney Diseases Grant 1R01DK080906-01A1 to Marian Tanofsky-Kraff, USUHS Grant R072IC to Marian Tanofsky-Kraff, and National Institute of Mental Health Grant K24MH070446 to Denise E. Wilfley.

Binge eating disorder (BED), presently considered a form of “eating disorder not otherwise specified” (American Psychiatric Association [APA], 2000), is characterized by recurrent episodes of binge eating without regular compensatory behaviors. Binge episodes are defined as the consumption of a large amount of food during which a sense of loss of control (LOC) over eating is experienced (APA, 2000). BED is common among obese adults and is associated with dysfunctional eating attitudes, marked psychiatric distress (Wilfley, Wilson, & Agras, 2003), and impairments in physical health (Johnson, Spitzer, & Williams, 2001). With the inclusion of BED under consideration for the next edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM), establishing the clinical relevance of early eating patterns that may precede the development of the disorder is warranted.

Very limited longitudinal research has examined the precursors of BED. However, prospective studies in adolescent samples have identified some predictors of eating disorders that include binge eating in their symptomatology (Killen et al., 1996, 1994; McKnight Investigators, 2003). Three prospective studies of adolescent girls who did not carry a diagnosis of a subclinical or clinical eating disorder but who scored highly on measures of “weight concern” (Killen et al., 1996, 1994) and thin body preoccupation (McKnight Investigators, 2003) were found to be at high risk for developing partial- or full-syndrome eating disorders. A fourth study that examined overweight adolescent boys and girls replicated the finding that weight importance predicted disordered eating (Neumark-Sztainer, Wall, Story, & Sherwood, 2009). In this study, binge eating at baseline was not predictive of increases in disordered eating, but the study did not specifically examine the development of partial- or full-syndrome eating disorders. To our knowledge, no study has prospectively examined the development of disordered eating, including the development of BED, in younger children. Studying children prior to adolescence is especially warranted because the emergence of disordered eating behavior may begin as early as middle childhood (Tanofsky-Kraff, 2008).

Although youth often report binge eating, it is typically with less frequency than required to meet criteria for BED (Tanofsky-Kraff, 2008). Traditional interview assessments of binge eating behavior require that a binge episode be diagnosed only if the amount of food consumed is deemed “unambiguously large” (Bryant-Waugh, Cooper, Taylor, & Lask, 1996; Fairburn & Cooper, 1993). Given the varying energy needs of physically developing boys and girls, it is often difficult to make this determination for children of different ages. For example, the consumption of an entire large pizza by a child or adolescent of any age would likely be considered unambiguously large. By contrast, an amount of five slices of pizza eaten by a 16-year-old boy might be less clear and thus deemed an ambiguously large amount of food that, even if accompanied by a sense of LOC over eating, might not be classified as an objective binge eating episode. However, the experience of LOC over eating, regardless of whether the reported amount of food consumed is considered unambiguously or ambiguously large, is common in youth (Tanofsky-Kraff, 2008). Thus, the term LOC eating, as opposed to binge eating, is often adopted when working with children and adolescents in order to be inclusive of all episodes involving LOC (Shomaker et al., 2009).

Most of the existing pediatric literature describes children who report only one episode of LOC eating in the month prior to assessment or even less frequent episodes (Tanofsky-Kraff, 2008). In spite of its low frequency, LOC eating in children is associated with greater body mass index (BMI; kg/m²) and body fat mass as well as greater psychological distress compared with youth without such behaviors (Tanofsky-Kraff, 2008). Both reported binge (Field et al., 2003; Stice, Cameron, Killen, Hayward, & Taylor, 1999; Tanofsky-Kraff et al., 2006) and LOC (Tanofsky-Kraff, Yanovski, et al., 2009) eating in youth predict excessive body weight gain in longitudinal studies of children and adolescents. Although infrequent LOC episodes in young children might be anticipated to be precursors to exacerbated disordered eating, this supposition has not been well tested. None of the aforementioned prospective studies of eating disorders has examined LOC eating as a predictor (Killen et al., 1996, 1994; McKnight Investigators, 2003; Neumark-Sztainer et al., 2009).

Few theoretical models exist to describe the development of exacerbated disordered eating in young children. Adult theories suggest that the development and maintenance of binge eating episodes are tied to negative affect (e.g., Heatherton & Baumeister, 1991; Leon, Fulkerson, Perry, & Early-Zald, 1995). Binge eating may temporarily reduce momentary negative affective states by serving as a maladaptive coping strategy (Arnow, Kenardy, & Agras, 1992) or as an “escape” from self-awareness (Heatherton & Baumeister, 1991). Yet, a pernicious cycle is proposed to develop in which binge eating ultimately promotes worsening of mood (Barker, Williams, & Galambios, 2006). Longitudinal data examining the development of depression among adolescent girls indicate that symptoms of bulimia nervosa, including binge eating, predict the onset of major depression (Stice, Hayward, Cameron, Killen, & Taylor, 2000). Additionally, adolescent girls' depressive symptoms and binge eating may interact cyclically to maintain binge eating behaviors (Presnell, Stice, Seidel, & Madeley, 2009).

Although the temporal relationship between LOC eating and negative affect has not been well studied in youth, it might be expected that preadolescent children's LOC eating would precede and predict the development of exacerbated disordered eating and negative affect. Whereas the onset of LOC eating episodes are often reported during childhood (Tanofsky-Kraff, 2008), clinically relevant shape, weight, and eating concerns, as well as dietary restraint and depressive symptoms, typically do not emerge until adolescence (Lewinsohn, Rohde, Seeley, Klein, & Gotlib, 2000; Stice, Killen, Hayward, & Taylor, 1998). Although self-reports of negative moods states among non-treatment-seeking children with LOC eating patterns are consistently higher than among youth without LOC, scores are typically well below clinically significant levels (Tanofsky-Kraff, 2008). Children with LOC often report an experience of “numbing”—a feeling that they are unaware of what is going on in the moment—during LOC episodes (Tanofsky-Kraff et al., 2007), suggesting that children with LOC may have difficulty describing emotional states and that LOC behavior may evolve into serving a similar affective coping function as has been described among adults (Arnow et al., 1992; Heatherton & Baumeister, 1991). Likewise, the state negative affect that often ensues from children's LOC eating episodes (Tanofsky-Kraff et al., 2007; Tanofsky-Kraff, Marcus, Yanovski, & Yanovski, 2008) could be expected to promote affective distress and worsening of disordered eating attitudes and behaviors as youth enter adolescence.

We therefore tested a number of hypotheses about the prospective relationships among childhood LOC, disordered eating (restraint and eating, shape, and weight concerns), and negative affect (measured as depressive and anxiety symptoms). Our primary hypothesis was that childhood LOC eating would predict increased disordered eating and negative affect in adolescence. Furthermore, we expected that adolescents whose childhood LOC persisted (i.e., who reported LOC eating at both baseline and follow-up) would experience the poorest psychosocial functioning at follow-up, compared with those who never had LOC or whose childhood LOC resolved (i.e., who reported LOC episodes only at baseline). Finally, although the prevalence of eating disorders is relatively low during early adolescence (Lewinsohn, Striegel-Moore, & Seeley, 2000; Stice, Presnell, & Bearman, 2001), we investigated LOC eating among middle childhood youth as a precursor of partial- and full-syndrome BED during adolescence.

Participants

A non-treatment-seeking community sample of overweight and nonoverweight children (age 6–13 years) was studied between July 1999 and August 2009. By design, the sample was enriched for overweight children. Participants were recruited through two waves of notices mailed to first- through fifth-grade children in the Montgomery County and Prince George's County, Maryland, school districts; advertisements in local newspapers; and two mailings to local family physicians and pediatricians. Mailings to families and physicians requested participation of children willing to undergo phlebotomy (multiple blood draws) and imaging assessments (i.e., magnetic resonance imaging, pelvic ultrasound for girls, wrist x-ray for determination of bone age) for studies investigating hormones and metabolic functioning in children. Mailings also specified that no treatment would be offered. In addition to the questionnaire and interview assessments, for those families that agreed to participate, children also underwent air displacement plethysmography (Life Measurement Inc., Concord, CA), bioelectrical impedance, skinfold thickness to determine body composition, various measures of insulin sensitivity, repeated urine collections, and a medical history and a physical examination that involved determination of pubertal stage by a pediatric endocrinologist or nurse practitioner. Children's compensation ranged from $70 to $170 per visit depending on their level of participation, with the higher amount for the full assessment panel. All understood that they would not receive treatment as part of the study but would be financially compensated for their participation. Subjects were healthy and medication-free for at least 2 weeks prior to baseline evaluation. Children provided written assent and parents gave written consent for participation in the protocol. This study was approved by the Eunice Kennedy Shriver National Institute of Child Health and Human Development institutional review board.

Procedures

At baseline, the Eating Disorder Examination (Version 12OD/C.2; EDE; Fairburn & Cooper, 1993) adapted for children (Bryant-Waugh et al., 1996) was administered to each participant to determine the presence or absence of LOC eating, as described previously (Tanofsky-Kraff et al., 2007, 2004). On the basis of their responses to the child EDE, participants were categorized as engaging in objective binge episodes (unambiguous overeating with LOC), subjective binge episodes (LOC with ambiguous overeating or without overeating), objective overeating (overeating without LOC), or no episodes (normal meals involving neither LOC nor overeating) over the 28 days prior to assessment. As described previously (Tanofsky-Kraff, Faden, Yanovski, Wilfley, & Yanovski, 2005), all children were queried as to whether they had ever experienced LOC over eating (reporting at least one instance of an objective or a subjective binge episode ever, that is, LOC ever). The EDE generates four subscales—restraint (cognitive and behavioral dietary restraint), eating concern, shape concern, and weight concern—as well as a global score. These continuous variables were used as measures of children's disordered eating attitudes. Variables generating the subscales are independent of those identifying eating episodes (Tanofsky-Kraff et al., 2004). Tests of the EDE adapted for children have demonstrated good interrater reliability (Spearman rank correlations from .91 to 1.00) and discriminant validity in eating disordered samples and matched controls age 8–14 years (Christie, Watkins, & Lask, 2000). Among nonoverweight and overweight 6- to 13-year-olds, the child version of the EDE revealed excellent interrater reliability, with a Cohen's kappa for presence of the different eating episode categories of 1.00 (p < .001; Tanofsky-Kraff et al., 2004). The child version differs from the adult EDE only in that its script has been edited to make it more accessible to children age 8–14 years. Both versions generate the same eating episodes and subscales. In a sample including a broad age spectrum (8–18 years), the child and adult EDEs have been successfully combined (Tanofsky-Kraff et al., 2007). The EDE has good interrater reliability for all episode types (Spearman correlation coefficients: ≥ .70; Rizvi, Peterson, Crow, & Agras, 2000).

Data collection at follow-up was identical to the baseline assessment other than the adult EDE (Fairburn & Cooper, 1993), as opposed to the child version, was administered to all returning participants. Although children were seen annually for physical assessments, the EDE was administered only at the baseline and follow-up visits.

In addition, the Standard Pediatric Eating Episode Interview (Tanofsky-Kraff et al., 2007) was administered following the overeating section of the EDE to assess the contextual, behavioral, physical, and emotional aspects of aberrant eating episodes, including the associated features of binge episodes as defined by the DSM (4th ed., text rev.; DSM–IV–TR; APA, 2000). The Standard Pediatric Eating Episode Interview is designed to distinguish eating episodes with LOC from those without LOC (Tanofsky-Kraff et al., 2007).

Partial-syndrome BED was defined as reports of fewer than eight episodes per month for 6 months of LOC eating involving unambiguously large amounts of food (full-syndrome BED) but at least four episodes of LOC eating involving equivocally large and/or unambiguously large amounts of food on average per month, for at least 3 months. For partial- or full-syndrome BED, LOC episodes were characterized by at least three DSM–IV–TR-associated features of binge eating episodes (e.g., eating more rapidly than normal, eating when not physically hungry; APA, 2000). Although consistent with one study of children and adolescents (Tanofsky-Kraff et al., 2007), this definition is more conservative than that of most adult studies that have typically used a frequency criterion of at least one binge or LOC episode per month (Crow, Agras, Halmi, Mitchell, & Kraemer, 2002; Striegel-Moore et al., 2000).

At baseline and follow-up, participants completed the Children's Depression Inventory, a reliable and well-validated 27-item self-rated measure of depressive symptoms (Kovacs, 1982). Internal consistency reliability in this widely used measure is good, with coefficients ranging from .80 to .87 across samples of non-treatment-seeking community youth and children in school settings, age 7–17 years (Kovacs, 1982; Ollendick & Yule, 1990; Saylor, Finch, Spirito, & Bennett, 1984; Smucker, Craighead, Craighead, & Green, 1986). The total score was used. In the present sample, the Children's Depression Inventory demonstrated very good temporal stability (intraclass correlation = .41, p = .001). Children also completed the State–Trait Anxiety Inventory for Children A–Trait Scale, a 20-item self-report measure of trait anxiety that is widely used and psychometrically sound (Spielberger, Gorsuch, Lushene, Vagg, & Jacobs, 1983). Internal consistency for this well-validated questionnaire is very good, with correlations ranging from .78 to .91 in samples of elementary school and community children, none of whom were seeking psychiatric or medical treatment (Muris, Merckelbach, Ollendick, King, & Bogie, 2002; Papay & Spielberger, 1986; Spielberger et al., 1983). In the present sample, the measure demonstrated good temporal stability (intraclass correlation = .39, p = .002).

Height and weight were measured, and BMI was calculated, as previously described (Tanofsky-Kraff et al., 2004).

Statistical Analysis

Analyses were conducted with SPSS 16.0 or SAS 8.0. Skew and kurtosis were satisfactory on all variables, and outliers were adjusted to fall 1.5 times the interquartile range below or above the 25th or 75th percentile (Behrens, 1997). This strategy was used because it minimizes outliers' influence on the characteristics of the distribution, minimally changes the distribution overall, and avoids potential bias associated with eliminating outliers altogether. Demographic characteristics for those with missing data were examined with independent-samples t tests or chi-square analyses to test differences between children who did and did not complete a follow-up assessment. Missing data were imputed with the multiple imputation procedure in SAS. The missing data model included demographic variables of age (years), sex (male vs. female), race/ethnicity (non-Hispanic Caucasian vs. other), socioeconomic status (Hollingshead, 1975), and puberty according to the stages of Tanner (Marshall & Tanner, 1969, 1970); years in study; both baseline and follow-up values of BMI; disordered eating attitudes (restraint, eating concern, shape concern, and weight concern subscales and global score); LOC eating (presence vs. absence); and symptoms of depression and anxiety. Because of the large missing data fraction, 10 imputed data sets were produced (Allison, 2009; Schafer & Graham, 2002). Following standard procedures for multiple imputation, each data set was analyzed separately, and the results for the 10 data sets were combined via the MIANALYZE procedure in SAS.

Independent-samples t tests were used to compare youth with and without baseline LOC ever (presence vs. absence) on demographic characteristics, baseline and follow-up disordered eating, and depressive and anxiety symptoms. A series of analyses of covariance were conducted to examine whether the independent variable of baseline LOC ever (presence vs. absence) predicted the dependent variables of follow-up disordered eating attitudes (global, restraint, eating concern, shape concern, and weight concern), symptoms of depression, and symptoms of anxiety. We examined both “simple” adjusted models that accounted for years between baseline and follow-up and the respective baseline symptom (disordered eating, depressive symptoms, or anxiety symptoms), and models that adjusted for these covariates as well as baseline age (years), sex (male vs. female), race/ethnicity (non-Hispanic Caucasian vs. other), and BMI change between baseline and follow-up.

Logistic regression was used to describe the unadjusted odds of the dependent variable of LOC at follow-up (presence vs. absence) based on the independent variable of baseline LOC ever status (presence vs. absence). Odds of LOC at follow-up based on baseline LOC ever were also examined after adjusting for years between baseline and follow-up. Change in number of LOC eating episodes in the past month (defined continuously) between baseline and follow-up were described with paired-samples t tests among the whole sample, as well as just among children who reported persistent LOC eating (present at both baseline and follow-up). Repeated measures analysis of variance was used to adjust these comparisons for time between baseline and follow-up. Another series of analyses of covariance were conducted to investigate whether the independent variable of LOC persistence predicted the dependent variables of follow-up disordered eating attitudes (global, restraint, eating concern, shape concern, and weight concern), depressive symptoms, and anxiety symptoms. LOC persistence was defined categorically as (a) no LOC at baseline or follow-up (“never LOC”); (b) LOC ever at baseline, no LOC at follow-up (“resolved”); (c) no LOC ever at baseline, LOC at follow-up (“emergent”); and (d) LOC at both baseline and follow-up (“persistent”). Simple models were examined adjusting only for years between baseline and follow-up and the respective baseline symptom (disordered eating, depressive symptoms, or anxiety symptoms), and we also examined models that adjusted for these covariates as well as baseline age (years), sex (male vs. female), race/ethnicity (non-Hispanic Caucasian vs. other), and BMI change between baseline and follow-up. As recommended (Cohen, 1990; Saville, 1990), two-tailed, least squares difference tests were used to follow up on all pairwise comparisons between groups.

A logistic regression was conducted to assess predictors of the dependent variable of follow-up partial- or full-syndrome BED onset. The independent variables included LOC ever (presence vs. absence), disordered eating attitudes, and depressive and anxiety symptoms as predictors. The sets of covariates considered were years between baseline and follow-up alone, plus years between baseline and follow-up, baseline age (years), sex (male vs. female), race/ethnicity (non-Hispanic Caucasian vs. other), and BMI change between baseline and follow-up.

One hundred ninety-five children (age 6–13 years) were seen for a baseline visit. Baseline data for 162 participants have been reported elsewhere (Tanofsky-Kraff et al., 2005, 2004). Of the 195 study participants, 118 (60.5%) completed a follow-up assessment an average of 4.7 years (SD = 1.2; range: 2.6–7.1) later. Compared with those who did not return for a follow-up assessment, youth who completed a follow-up visit were older at baseline (M = 10.38, SD = 1.49 vs. M = 9.63, SD = 1.63, p = .01) but did not significantly differ in sex, race, or baseline BMI; prevalence of LOC eating; disordered eating; or depressive or anxiety symptoms. As described above, data were imputed for the 77 children who did not return for a follow-up assessment, and combined results from the imputed data are presented.

At baseline, no child met full- or partial-syndrome criteria for a DSM–IV–TR eating disorder. Forty-six children (23.6%) reported having experienced LOC eating at least once in their lifetime. Of these participants, approximately half (n = 22, 47.8%) experienced at least one recent LOC episode in the month prior to assessment. In terms of the types and number of episodes in the past month, nine children reported only objective binge episodes (eight reported one episode, and one reported four episodes), 11 reported only subjective binge episodes (four children reported one episode, five reported two episodes, one reported three episodes, and one reported four episodes), and two reported one objective binge episode and one subjective binge episode in the prior month. In line with previously published data (Tanofsky-Kraff et al., 2005), youth with baseline LOC ever had significantly higher baseline BMI, BMI z scores (Kuczmarski et al., 2002), disordered eating attitudes, and depressive symptoms (ps < .01; see Table 1) compared with those who never experienced LOC. Compared with those without baseline LOC, youth with baseline LOC ever were also significantly younger at follow-up (p = .04) and continued to have significantly higher follow-up BMI, BMI z scores (Kuczmarski et al., 2002), and EDE global scores as well as higher restraint and shape and weight concern subscale scores (ps < .01; see Table 2).

Participant Characteristics at Baseline

Participant Characteristics at Follow-Up

Baseline LOC Eating as a Predictor of Follow-Up Disordered Eating and Negative Affect

In simple models adjusting only for years between baseline and follow-up and the respective baseline subscale score, baseline LOC ever was associated with greater follow-up EDE global, restraint, and shape and weight concern (ps < .02). Even after adjusting for additional covariates (baseline age, sex, race/ethnicity, and BMI change), baseline LOC ever predicted greater follow-up EDE global, restraint, and shape and weight concern scores (ps ≤ .04; see Figure 1). Baseline LOC ever significantly predicted follow-up anxiety in both the simple model adjusting for only baseline anxiety and years between baseline and follow-up (p = .03) and a model adjusting for all covariates (p = .05). Baseline LOC ever did not predict follow-up eating concern or depressive symptoms in either model (ps > .48).

Figure 1. Baseline loss of control eating and follow-up disordered eating attitudes. Experiencing loss of control over eating ever at baseline predicted increases on the Eating Disorder Examination restraint, shape concern, and weight concern subscales at follow-up. Analysis adjusted for baseline age, sex, race, years in study, body mass index growth, and respective baseline disordered eating subscale. Error bars indicate one standard error above and below the mean. N = 195, ps ≤ .04.

Descriptive Information on Persistence of LOC Eating

The presence of LOC ever at baseline was associated with more than a twofold greater likelihood of reported LOC at follow-up (OR = 2.35, 95% CI [1.10, 5.01], p = .03). Among those reporting LOC ever at baseline (n = 46), 52.2% (n = 24) reported persistent LOC eating in the month prior to follow-up assessment. In contrast, among those who did not report LOC ever at baseline (n = 149), only 30.9% (n = 46) reported emergent LOC at follow-up; the majority of this group (69.1%; n = 103) never reported LOC at either time. Of those endorsing LOC ever at baseline, 47.8% (n = 22) were resolved at follow-up. After adjusting for years between baseline and follow-up, baseline LOC ever remained a significant predictor of follow-up LOC (OR = 2.67, 95% CI [1.15, 6.22], p = .02). When persistence was characterized in terms of number of LOC episodes in the past month, LOC episodes per month for the whole sample increased from baseline (M = 0.19, SE = 0.04) to follow-up (M = 0.53, SE = 0.16, p = .04). Similarly, among the 24 youth who endorsed baseline LOC ever and follow-up LOC, the average number of LOC episodes significantly increased by 1.33 (SE = 0.53) episodes per month between baseline (M = 0.79, SE = 0.24) and follow-up (M = 2.12, SE = 0.40, p = .01). Significant increases in number of LOC episodes continued to be observed after accounting for years between baseline and follow-up, both in the whole sample and just among those with persistent LOC (ps < .04).

Persistence of LOC Eating as a Predictor of Disordered Eating and Negative Affect

In simple models (only covariates of years between baseline and follow-up and respective baseline subscale) and in models accounting for all covariates, there were significant overall effects for LOC persistence (never LOC, resolved, emergent, persistent) for all four follow-up EDE subscales (all main effects ps < .004): global score in model adjusting for all covariates, F(3, 143) = 12.12, p < .001 (see Figure 2A). In simple adjusted models, pairwise comparisons indicated that youth with persistent LOC had significantly higher scores than the never LOC group for EDE global score and all four EDE subscales (ps < .006); higher scores than the resolved group on EDE global score, eating concern, and shape concern (ps < .02); and higher scores than the emergent LOC group on EDE global score (p = .04). Youth with emergent LOC had higher scores on EDE global and all four subscales compared with the never LOC group (ps < .02), and emergent LOC youth also had higher EDE eating concern than youth with resolved LOC (p = .02). These significant differences remained after accounting for all covariates (ps < .05).

Figure 2. (A) Persistent loss of control (LOC) eating and follow-up disordered eating. Eating Disorder Examination global score at follow-up is shown. Children who reported ever experiencing LOC at baseline and LOC at follow-up (persistent) had significantly greater disordered eating at follow-up compared with youth who never reported LOC (never LOC), those who reported LOC at baseline only (resolved), and those who reported LOC at follow-up (emergent). Analysis adjusted for baseline age, sex, race, years in study, body mass index growth, and baseline global disordered eating. N = 195. Main effect p < .001. (B) Persistent LOC eating and follow-up depressive symptoms. Children who reported ever experiencing LOC at baseline and LOC at follow-up (persistent) had significantly greater total scores on the Children's Depression Inventory at follow-up compared with youth who never reported LOC (never LOC) and those who reported LOC at baseline only (resolved). Analysis adjusted for baseline age, sex, race, years in study, body mass index growth, and baseline depressive symptoms. Error bars indicate one standard error above and below the mean. N = 195. Main effect p = .03.

In a simple adjusted model and a model accounting for all covariates, there was an overall main effect of LOC persistence on symptoms of depression: in model adjusting for all covariates, F(3, 262) = 3.11, p = .03 (see Figure 2B). In a simple adjusted model, pairwise comparisons indicated that persistent LOC youth had higher follow-up depressive symptoms than the never LOC group (p = .03) and the resolved LOC group (p = .01). The same significant differences were observed when accounting for all covariates (ps < .05). LOC persistence showed nonsignificant trends for association with follow-up symptoms of anxiety both in the simple adjusted model, F(3, 311) = 2.44, p = .06, and after adjusting for all covariates, F(3, 300) = 2.28, p = .08.

Development of Partial- or Full-Syndrome BED

At follow-up, nine (4.5%) participants met partial- or full-syndrome criteria for BED (LOC episodes in the month prior to assessment: M = 5.7, SE = 1.3; range: 4–15). A subset (60%) of EDEs and Standard Pediatric Eating Episode Interview for these participants were taped and corated to examine interrater reliability for presence of the partial- and full-syndrome BED diagnosis. Cohen's kappa for the identification of partial- or full-syndrome BED was 1.00 (p < .001). In the model examining the development of partial- or full-syndrome BED at follow-up, only baseline reports of LOC ever served as a significant contributor (OR = 10.8, 95% CI [1.3, 88.1], p = .03), after accounting for all other variables in the model including BMI change, baseline symptoms of depression and anxiety, and all four EDE subscales at baseline. Because the sample of participants who developed BED was small, yielding a very wide confidence interval for the predictive value of LOC ever, we reanalyzed the model removing all nonsignificant variables except years in the study (sex, race, baseline age, depressive and anxiety symptoms, and disordered eating, BMI growth). Baseline reports of LOC ever continued to serve as a significant predictor of BED (OR = 5.07, 95% CI [1.1, 24.5], p = .04). This analysis suggested that youth who reported ever having experienced LOC at baseline were greater than 5 times more likely to develop partial- or full-syndrome BED at follow-up.

Follow-Up Exploratory Analyses

Secondary analyses were conducted to examine predictors of LOC onset at follow-up. To examine predictors of the dependent variable of follow-up LOC eating onset (presence vs. absence), we conducted binary logistic regression analyses with only the subset of children who did not endorse baseline LOC ever (n = 149). The independent variables were baseline disordered eating attitudes (global, restraint, shape concern, and weight concern), depressive symptoms, and anxiety symptoms. The sets of covariates considered were years between baseline and follow-up alone, and then years between visits and baseline age (years), sex (male vs. female), race/ethnicity (non-Hispanic Caucasian vs. other), and BMI change between baseline and follow-up. In the subset of children who reported no LOC ever at baseline, neither disordered eating attitudes nor symptoms of depression or anxiety predicted follow-up emergent LOC eating in any model (ps > .28).

To further explore the nature of the significant relationships observed between persistence of LOC and increases in disordered eating and negative affect, we conducted a series of follow-up linear multiple regressions regressing the dependent variables of change scores (follow-up minus baseline) in disordered eating (EDE global score, restraint, eating concern, shape concern, and weight concern subscales), depressive symptoms, and anxiety symptoms on the independent variables of change score in LOC eating episodes in the past month. In both simple adjusted models accounting only for years between baseline and follow-up and models accounting for all covariates, there was no significant association between change in number of LOC episodes and any of the dependent variables (all ps > .50).

Using the Eating Disorder Examination, a well-accepted interview assessment method for identification of eating disorders, we found, among a sample of non-treatment-seeking children age 6–13 years who were reassessed approximately 5 years later, that those who reported having ever experienced loss of control over their eating at baseline were significantly more likely to develop partial- or full-syndrome binge eating disorder than children who had never experienced LOC. The presence of reported LOC eating at baseline also predicted increases in disordered eating attitudes and symptoms of anxiety at follow-up but not in depressive symptoms. However, children with reported LOC eating at both baseline and follow-up experienced the greatest disordered eating within all domains and the greatest increases in symptoms of depression.

The finding that childhood LOC is a significant indicator for the development of persistent or worsened disordered eating attitudes and anxiety symptoms almost 5 years later is novel and may have important clinical implications. LOC eating has been consistently associated in cross-sectional studies with increased psychological distress (Tanofsky-Kraff, 2008) and found to be predictive of excess body weight gain in young children (Tanofsky-Kraff, Yanovski, et al., 2009). Our data suggest that infrequent reports of LOC may also be a precursor for increased disordered eating and the development of partial- or full-syndrome BED. Some preliminary studies indicate that reducing binge and LOC eating may be effective for both weight loss (Jones et al., 2008) and obesity prevention (Tanofsky-Kraff, Wilfley, et al., 2009) in youth. It remains to be determined whether such interventions may simultaneously prevent the development of eating disorders.

Almost 5% of the sample developed partial- or full-syndrome BED at follow-up. Although there are limited data on the prevalence of BED in early adolescence, rates for LOC eating, typically defined as one episode in the month prior to assessment or even less frequent LOC episodes, range from approximately 2% to 40%, with higher estimates among weight-loss-treatment-seeking (vs. community) samples and higher prevalence among adolescents (vs. children) and in studies using questionnaires rather than diagnostic interviews (Tanofsky-Kraff, 2008). In adult samples, the rate of DSM–IV–TR BED is approximately 3% in the general population (Hudson, Hiripi, Pope, & Kessler, 2007), with higher estimates among individuals seeking weight-loss treatment (de Zwaan, 2001). Although our sample was not seeking weight loss treatment, it differed from other pediatric community samples because it was enriched for overweight youth, which likely contributed to the relatively higher rate of partial- and full-syndrome BED found at follow-up.

It is notable that only LOC, as opposed to weight, shape, and eating concerns, or negative affect, was a clinically relevant behavior for the development of BED. This finding is in contrast to studies that examined the development of any eating disorder (i.e., not exclusively BED) during adolescence, where body shape and weight concerns were important predictors for later eating disorders (Killen et al., 1996, 1994; McKnight Investigators, 2003). Although the aforementioned studies included different covariates than those included in the present analyses, our findings generally did not differ whether we conducted simple analyses adjusting only for years to follow-up and the respective baseline variable or analyses including all covariates. It should be noted that these investigations did not examine baseline LOC eating as a predictor. Our results may be related to the developmental differences in emotional and cognitive constructs that are found during middle childhood versus adolescence. With regard to symptoms of depression and anxiety, it is possible that at baseline the children in our sample had yet to develop symptoms of mood problems. Given that our sample consisted of non-treatment-seeking children without a current eating disorder, depressive symptoms and trait anxiety may not have been a common component of their baseline psychological status. Indeed, the baseline scores reported by the sample were well below clinically significant cutoffs (Kazdin, Colbus, & Rodgers, 1986; Lobovits & Handal, 1985; Spielberger et al., 1983). LOC eating may conceivably be an early behavioral marker preceding, or even possibly masking, disordered eating attitudes and negative affect. In a multisite study of children and adolescents, the experience of numbing was highly correlated with LOC eating episodes (Tanofsky-Kraff et al., 2007). These data point to another possibility; namely, that children reporting LOC may be less aware of their emotional experiences than of their actual behaviors. Both suppositions may potentially support “escape theory,” which views binge eating as a motivated attempt to escape aversive self-awareness or emotional distress (Heatherton & Baumeister, 1991). Regardless, during middle childhood, reported eating patterns may thus be more salient in predicting later exacerbated disordered eating in adolescence than reports of either body weight or mood-related distress.

Children who reported having engaged in LOC eating even one time at baseline were at greater risk for the development of increased shape and weight concerns at follow-up than those who did not report such episodes. Notably, overconcern with body shape and weight is a key feature across the eating disorder diagnoses (Wilfley, Schwartz, Spurrell, & Fairburn, 2000) and is associated with greater impairment when present in individuals with BED (Grilo, Masheb, & White, 2010). However, those youth who reported LOC at both baseline and follow-up were most at risk for exacerbated disordered eating and increases in depressive symptoms, with the average unadjusted Children's Depression Inventory total score nearing the clinical concern cutoff of 12 (Kazdin et al., 1986; Lobovits & Handal, 1985). These findings persisted even after accounting for the contribution of BMI growth, suggesting that the increased distress predicted by LOC cannot be attributed solely to excess weight gain. Coupled with data indicating that binge and LOC eating have been shown to predict excess weight and fat gain in youth (Field et al., 2003; Stice et al., 1999; Tanofsky-Kraff et al., 2006; Tanofsky-Kraff, Yanovski, et al., 2009), our findings support proposals that obesity and eating disorder interventions should be coordinated (Neumark-Sztainer et al., 2009; Yanovski, 2003).

Our findings from the present and prior (Tanofsky-Kraff, Yanovski, et al., 2009) studies suggest that in young children, the report of infrequent objective or subjective binge eating episodes that together constitute LOC eating is predictive of untoward results. Even quite rare LOC eating episodes appear to be associated with later adverse psychological outcomes. Although this may be the result of an overly conservative estimate of what constitutes a “large” amount of food in growing children, the experience of LOC eating may identify youth who are consuming more than they want to eat. It is possible that individuals with reported LOC have some disturbance in satiety signaling or reward activation pathways (Adam & Epel, 2007; Davis et al., 2008). The relationship between LOC and eating in the absence of hunger has been documented (Tanofsky-Kraff, Ranzenhofer, et al., 2008). Further, data indicate that youth with LOC tend to consume highly palatable dessert and snack-type foods (Hilbert, Tuschen-Caffier, & Czaja, 2010; Tanofsky-Kraff, McDuffie, et al., 2009; Theim et al., 2007); thus, LOC behaviors may be related to variations in opioid (reward) pathways, stemming from genetic polymorphisms in opioid–dopamine receptor genes, as suggested in studies of adults with and without BED (Davis et al., 2008). Though little is known about the neural circuitry of LOC eating in children, there are neuroimaging data in emotional eaters (Bohon, Stice, & Spoor, 2009) and individuals with BED (Schienle, Schäfer, Hermann, & Vaitl, 2009) to suggest that the processing of food-related stimuli may be fundamentally different in persons with and without aberrant eating patterns. Longitudinal data are needed to explore these potential neural pathways underlying LOC eating and the development of BED.

Strengths of this investigation include the prospective design, the use of a structured interview, and measured heights and weights. Limitations include the fact that children who reported LOC eating ever (i.e., prior to the EDE time frame) did not supply information on size of the meal consumed, the use of questionnaires to assess depressive and anxiety symptoms, and the relatively small sample at follow-up. Moreover, there was only one EDE follow-up assessment, which precluded an examination of the average duration of LOC eating during the course of the follow-up period. Because of this limitation, we are also unable to explore why approximately 48% of those with baseline LOC eating no longer reported such behaviors at follow-up. In addition, despite the fact that children were not seeking treatment, they were not recruited in a population-based fashion, were enriched for overweight, and had to travel to the National Institutes of Health Clinical Research Center to participate. These factors likely limit the generalizability of our findings to young children willing to participate in research that involves both physical and psychological measurements. Indeed, studying young children with various physical assessments, especially those that are often perceived as invasive (i.e., phlebotomy), likely renders a sample that is not reflective of the general population. Nevertheless, families were recruited for metabolic studies and understood that they would not receive treatment, and neither parents nor children had prior knowledge that they would be asked about disordered eating behaviors and attitudes.

In conclusion, among a nontreatment sample of overweight and nonoverweight middle-childhood youth, those who reported having engaged in LOC eating are more likely to develop worsening disordered eating. Moreover, those whose LOC eating persists over time appear to be at higher risk for increases in symptoms of depression. Future investigation is necessary to determine whether interventions aimed at reducing LOC eating during middle childhood are efficacious in the prevention of both eating disorders and excessive weight gain.

Adam, T. C., & Epel, E. S. (2007). Stress, eating and the reward system. Physiology and Behavior, 91(4), 449–458. doi: 10.1016/j.physbeh.2007.04.011

Allison, P. D. (2009). Missing data. In R. E.Millsap & A.Maydeu-Olivares (Eds.), The Sage handbook of quantitative methods in psychology (pp. 72–89). Thousand Oaks, CA: Sage.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

Arnow, B., Kenardy, J., & Agras, W. S. (1992). Binge eating among the obese: A descriptive study. Journal of Behavioral Medicine, 15(2), 155–170. doi:10.1007/BF00848323

Barker, E. T., Williams, R. L., & Galambios, N. L. (2006). Daily spillover to and from binge eating in first-year university females. Eating Disorders, 14(3), 229–242. doi: 10.1080/10640260600639079

Behrens, J. T. (1997). Principles and procedures of exploratory data analysis. Psychological Methods, 2(2), 131–160. doi: 10.1037/1082-989X.2.2.131

Bohon, C., Stice, E., & Spoor, S. (2009). Female emotional eaters show abnormalities in consummatory and anticipatory food reward: A functional magnetic resonance imaging study. International Journal of Eating Disorders, 42(3), 210–221. doi: 10.1002/eat.20615

Bryant-Waugh, R. J., Cooper, P. J., Taylor, C. L., & Lask, B. D. (1996). The use of the eating disorder examination with children: A pilot study. International Journal of Eating Disorders, 19(4), 391–397. doi: 10.1002/(SICI)1098-108X(199605)19:4<391::AID-EAT6>3.0.CO;2-G

Christie, D., Watkins, B., & Lask, B. (2000). Assessment. In B.Lask & R. J.Bryant-Waugh (Eds.), Anorexia nervosa and related eating disorders in childhood and adolescence (2nd ed., pp. 105–125). Hove, England: Psychology Press.

Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12), 1304–1312. doi: 10.1037/0003-066X.45.12.1304

Crow, S. J., Agras, W. S., Halmi, K., Mitchell, J. E., & Kraemer, H. C. (2002). Full syndromal versus subthreshold anorexia nervosa, bulimia nervosa, and binge eating disorder: A multicenter study. International Journal of Eating Disorders, 32(3), 309–318. doi: 10.1002/eat.10088

Davis, C., Levitan, R. D., Kaplan, A. S., Carter, J., Reid, C., Curtis, C., . . .Kennedy, J. L. (2008). Reward sensitivity and the D2 dopamine receptor gene: A case-control study of binge eating disorder. Progress in Neuro-Psychopharmacology and Biological Psychiatry, 32(3), 620–628. doi: 10.1016/j.pnpbp.2007.09.024

de Zwaan, M. (2001). Binge eating disorder and obesity. International Journal of Obesity and Related Metabolic Disorders, 25(Suppl. 1), S51–S55. doi: 10.1038/sj.ijo.0801699

Fairburn, C. G., & Cooper, Z. (1993). The Eating Disorder Examination (12th edition). In C. G.Fairburn & G. T.Wilson (Eds.), Binge eating, nature, assessment and treatment (pp. 317–360). New York, NY: Guilford.

Field, A. E., Austin, S. B., Taylor, C. B., Malspeis, S., Rosner, B., Rockett, H. R., . . .Colditz, G. A. (2003). Relation between dieting and weight change among preadolescents and adolescents. Pediatrics, 112(4), 900–906. doi: 10.1542/peds.112.4.900

Grilo, C. M., Masheb, R. M., & White, M. A. (2010). Significance of overvaluation of shape/weight in binge-eating disorder: Comparative study with overweight and bulimia nervosa. Obesity, 18(3), 499–504.

Heatherton, T. F., & Baumeister, R. F. (1991). Binge eating as escape from self-awareness. Psychological Bulletin, 110(1), 86–108. doi: 10.1037/0033-2909.110.1.86

Hilbert, A., Tuschen-Caffier, B., & Czaja, J. (2010). Eating behavior and familial interactions of children with loss of control eating: A laboratory test meal study. American Journal of Clinical Nutrition, 91(3), 510–518. doi: 10.3945/ajcn.2009.28843

Hollingshead, A. B. (1975). Four Factor Index of Social Status. Unpublished manuscript, Department of Sociology, Yale University, New Haven, CT.

Hudson, J. I., Hiripi, E., Pope, H. G., Jr., & Kessler, R. C. (2007). The prevalence and correlates of eating disorders in the National Comorbidity Survey Replication. Biological Psychiatry, 61(3), 348–358. doi: 10.1016/j.biopsych.2006.03.040

Johnson, J. G., Spitzer, R. L., & Williams, J. B. (2001). Health problems, impairment and illnesses associated with bulimia nervosa and binge eating disorder among primary care and obstetric gynaecology patients. Psychological Medicine, 31(8), 1455–1466. doi: 10.1017/S0033291701004640

Jones, M., Luce, K. H., Osborne, M. I., Taylor, K., Cunning, D., Doyle, A. C., . . .Taylor, C. B. (2008). Randomized, controlled trial of an Internet-facilitated intervention for reducing binge eating and overweight in adolescents. Pediatrics, 121(3), 453–462. doi: 10.1542/peds.2007-1173

Kazdin, A. E., Colbus, D., & Rodgers, A. (1986). Assessment of depression and diagnosis of depressive disorder among psychiatrically disturbed children. Journal of Abnormal Child Psychology, 14(4), 499–515. doi: 10.1007/BF01260519

Killen, J. D., Taylor, C. B., Hayward, C., Haydel, K. F., Wilson, D. M., Hammer, L., . . .Strachowski, D. (1996). Weight concerns influence the development of eating disorders: A 4-year prospective study. Journal of Consulting and Clinical Psychology, 64(5), 936–940. doi: 10.1037/0022-006X.64.5.936

Killen, J. D., Taylor, C. B., Hayward, C., Wilson, D. M., Haydel, K. F., Hammer, L. D., . . .Kraemer, H. (1994). Pursuit of thinness and onset of eating disorder symptoms in a community sample of adolescent girls: A three-year prospective analysis. International Journal of Eating Disorders, 16(3), 227–238. doi: 10.1002/1098-108X(199411)16:3<227::AID-EAT2260160303>3.0.CO;2-L

Kovacs, M. (1982). The Children's Depression Inventory: A self-rated depression scale for school aged youngsters. Unpublished manuscript, University of Pittsburgh School of Medicine, Pittsburgh, PA.

Kuczmarski, R. J., Ogden, C. L., Guo, S. S., Grummer-Strawn, L. M., Flegal, K. M., Mei, Z., . . .Johnson, C. L. (2002). 2000 CDC growth charts for the United States: Methods and development. Vital and Health Statistics, 11(246), 1–190.

Leon, G. R., Fulkerson, J. A., Perry, C. L., & Early-Zald, M. B. (1995). Prospective analysis of personality and behavioral vulnerabilities and gender influences in the later development of disordered eating. Journal of Abnormal Psychology, 104(1), 140–149. doi: 10.1037/0021-843X.104.1.140

Lewinsohn, P. M., Rohde, P., Seeley, J. R., Klein, D. N., & Gotlib, I. H. (2000). Natural course of adolescent major depressive disorder in a community sample: Predictors of recurrence in young adults. American Journal of Psychiatry, 157(10), 1584–1591. doi: 10.1176/appi.ajp.157.10.1584

Lewinsohn, P. M., Striegel-Moore, R. H., & Seeley, J. R. (2000). Epidemiology and natural course of eating disorders in young women from adolescence to young adulthood. Journal of the American Academy of Child & Adolescent Psychiatry, 39(10), 1284–1292. doi: 10.1097/00004583-200010000-00016

Lobovits, D. A., & Handal, P. J. (1985). Childhood depression: Prevalence using DSM–III criteria and validity of parent and child depression scales. Journal of Pediatric Psychology, 10(1), 45–54. doi: 10.1093/jpepsy/10.1.45

Marshall, W. A., & Tanner, J. M. (1969). Variations in pattern of pubertal changes in girls. Archives of Disease in Childhood, 44(235), 291–303. doi: 10.1136/adc.44.235.291

Marshall, W. A., & Tanner, J. M. (1970). Variations in the pattern of pubertal changes in boys. Archives of Disease in Childhood, 45(235), 13–23. doi: 10.1136/adc.45.239.13

McKnight Investigators. (2003). Risk factors for the onset of eating disorders in adolescent girls: Results of the McKnight longitudinal risk factor study. American Journal of Psychiatry, 160(2), 248–254. doi: 10.1176/appi.ajp.160.2.248

Muris, P., Merckelbach, H., Ollendick, T., King, N., & Bogie, N. (2002). Three traditional and three new childhood anxiety questionnaires: Their reliability and validity in a normal adolescent sample. Behaviour Research and Therapy, 40(7), 753–772. doi: 10.1016/S0005-7967(01)00056-0

Neumark-Sztainer, D., Wall, M., Story, M., & Sherwood, N. E. (2009). Five-year longitudinal predictive factors for disordered eating in a population-based sample of overweight adolescents: Implications for prevention and treatment. International Journal of Eating Disorders, 42(7), 664–672. doi: 10.1002/eat.20733

Ollendick, T. H., & Yule, W. (1990). Depression in British and American children and its relation to anxiety and fear. Journal of Consulting and Clinical Psychology, 58(1), 126–129. doi: 10.1037/0022-006X.58.1.126

Papay, J. P., & Spielberger, C. D. (1986). Assessment of anxiety and achievement in kindergarten and first- and second-grade children. Journal of Abnormal Child Psychology, 14(2), 279–286. doi: 10.1007/BF00915446

Presnell, K., Stice, E., Seidel, A., & Madeley, M. C. (2009). Depression and eating pathology: Prospective reciprocal relations in adolescents. Clinical Psychology & Psychotherapy, 16(4), 357–365. doi: 10.1002/cpp.630

Rizvi, S. L., Peterson, C. B., Crow, S. J., & Agras, W. S. (2000). Test–retest reliability of the Eating Disorder Examination. International Journal of Eating Disorders, 28(3), 311–316. doi: 10.1002/1098-108X(200011)28:3<311::AID-EAT8>3.0.CO;2-K

Saville, D. J. (1990). Multiple comparison procedures: The practical solution. American Statistician, 44(2), 174–180. doi: 10.2307/2684163

Saylor, C. F., Finch, A. J., Jr., Spirito, A., & Bennett, B. (1984). The Children's Depression Inventory: A systematic evaluation of psychometric properties. Journal of Consulting and Clinical Psychology, 52(6), 955–967. doi: 10.1037/0022-006X.52.6.955

Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. doi: 10.1037/1082-989X.7.2.147

Schienle, A., Schäfer, A., Hermann, A., & Vaitl, D. (2009). Binge-eating disorder: Reward sensitivity and brain activation to images of food. Biological Psychiatry, 65(8), 654–661. doi: 10.1016/j.biopsych.2008.09.028

Shomaker, L. B., Tanofsky-Kraff, M., Elliott, C., Wolkoff, L. E., Columbo, K. M., Ranzenhofer, L. M., . . .Yanovski, J. A. (2009). Salience of loss of control for pediatric binge episodes: Does size really matter?International Journal of Eating Disorders. Advance online publication. doi: 10.1002/eat.20767

Smucker, M. R., Craighead, W. E., Craighead, L. W., & Green, B. J. (1986). Normative and reliability data for the Children's Depression Inventory. Journal of Abnormal Child Psychology, 14(1), 25–39. doi: 10.1007/BF00917219

Spielberger, C. D., Gorsuch, R. L., Lushene, R. E., Vagg, P. R., & Jacobs, G. A. (1983). Manual for the State–Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press.

Stice, E., Cameron, R. P., Killen, J. D., Hayward, C., & Taylor, C. B. (1999). Naturalistic weight-reduction efforts prospectively predict growth in relative weight and onset of obesity among female adolescents. Journal of Consulting and Clinical Psychology, 67(6), 967–974. doi: 10.1037/0022-006X.67.6.967

Stice, E., Hayward, C., Cameron, R. P., Killen, J. D., & Taylor, C. B. (2000). Body-image and eating disturbances predict onset of depression among female adolescents: A longitudinal study. Journal of Abnormal Psychology, 109(3), 438–444. doi: 10.1037/0021-843X.109.3.438

Stice, E., Killen, J. D., Hayward, C., & Taylor, C. B. (1998). Age of onset for binge eating and purging during late adolescence: A 4-year survival analysis. Journal of Abnormal Psychology, 107(4), 671–675. doi: 10.1037/0021-843X.107.4.671

Stice, E., Presnell, K., & Bearman, S. K. (2001). Relation of early menarche to depression, eating disorders, substance abuse, and comorbid psychopathology among adolescent girls. Developmental Psychology, 37(5), 608–619. doi: 10.1037/0012-1649.37.5.608

Striegel-Moore, R. H., Dohm, F. A., Solomon, E. E., Fairburn, C. G., Pike, K. M., & Wilfley, D. E. (2000). Subthreshold binge eating disorder. International Journal of Eating Disorders, 27(3), 270–278. doi: 10.1002/(SICI)1098-108X(200004)27:3<270::AID-EAT3>3.0.CO;2-1

Tanofsky-Kraff, M. (2008). Binge eating among children and adolescents. In E.Jelalian & R. G.Steele (Eds.), Handbook of childhood and adolescent obesity (pp. 43–59). New York, NY: Springer.

Tanofsky-Kraff, M., Cohen, M. L., Yanovski, S. Z., Cox, C., Theim, K. R., Keil, M., . . .Yanovski, J. A. (2006). A prospective study of psychological predictors of body fat gain among children at high risk for adult obesity. Pediatrics, 117(4), 1203–1209. doi: 10.1542/peds.2005-1329

Tanofsky-Kraff, M., Faden, D., Yanovski, S. Z., Wilfley, D. E., & Yanovski, J. A. (2005). The perceived onset of dieting and loss of control eating behaviors in overweight children. International Journal of Eating Disorders, 38(2), 112–122. doi: 10.1002/eat.20158

Tanofsky-Kraff, M., Goossens, L., Eddy, K. T., Ringham, R., Goldschmidt, A., Yanovski, S. Z., . . .Yanovski, J. A. (2007). A multisite investigation of binge eating behaviors in children and adolescents. Journal of Consulting and Clinical Psychology, 75(6), 901–913. doi: 10.1037/0022-006X.75.6.901

Tanofsky-Kraff, M., Marcus, M. D., Yanovski, S. Z., & Yanovski, J. A. (2008). Loss of control eating disorder in children age 12 years and younger: Proposed research criteria. Eating Behaviors, 9(3), 360–365. doi: 10.1016/j.eatbeh.2008.03.002

Tanofsky-Kraff, M., McDuffie, J. R., Yanovski, S. Z., Kozlosky, M., Schvey, N. A., Shomaker, L. B., . . .Yanovski, J. A. (2009). Laboratory assessment of the food intake of children and adolescents with loss of control eating. American Journal of Clinical Nutrition, 89(3), 738–745. doi: 10.3945/ajcn.2008.26886

Tanofsky-Kraff, M., Ranzenhofer, L. M., Yanovski, S. Z., Schvey, N. A., Faith, M., Gustafson, J., . . .Yanovski, J. A. (2008). Psychometric properties of a new questionnaire to assess eating in the absence of hunger in children and adolescents. Appetite, 51(1), 148–155. doi: 10.1016/j.appet.2008.01.001

Tanofsky-Kraff, M., Wilfley, D. E., Young, J. F., Mufson, L., Yanovski, S. Z., Glasofer, D. R., . . .Schvey, N. A. (2009). A pilot study of interpersonal psychotherapy for preventing excess weight gain in adolescent girls at-risk for obesity. International Journal of Eating Disorders. Advance online publication. doi: 10.1002/eat.20773

Tanofsky-Kraff, M., Yanovski, S. Z., Schvey, N. A., Olsen, C. H., Gustafson, J., & Yanovski, J. A. (2009). A prospective study of loss of control eating for body weight gain in children at high risk for adult obesity. International Journal of Eating Disorders, 42(1), 26–30. doi: 10.1002/eat.20580

Tanofsky-Kraff, M., Yanovski, S. Z., Wilfley, D. E., Marmarosh, C., Morgan, C. M., & Yanovski, J. A. (2004). Eating-disordered behaviors, body fat, and psychopathology in overweight and normal-weight children. Journal of Consulting and Clinical Psychology, 72(1), 53–61. doi: 10.1037/0022-006X.72.1.53

Theim, K. R., Tanofsky-Kraff, M., Salaita, C. G., Haynos, A. F., Mirch, M. C., Ranzenhofer, L. M., . . .Yanovski, J. A. (2007). Children's descriptions of the foods consumed during loss of control eating episodes. Eating Behaviors, 8(2), 258–265. doi: 10.1016/j.eatbeh.2006.10.001

Wilfley, D. E., Schwartz, M. B., Spurrell, E. B., & Fairburn, C. G. (2000). Using the Eating Disorder Examination to identify the specific psychopathology of binge eating disorder. International Journal of Eating Disorders, 27(3), 259–269. doi: 10.1002/(SICI)1098-108X(200004)27:3<259::AID-EAT2>3.0.CO;2-G

Wilfley, D. E., Wilson, G. T., & Agras, W. S. (2003). The clinical significance of binge eating disorder. International Journal of Eating Disorders, 34(Suppl.), S96–S106. doi: 10.1002/eat.10209

Yanovski, S. Z. (2003). Binge eating disorder and obesity in 2003: Could treating an eating disorder have a positive effect on the obesity epidemic?International Journal of Eating Disorders, 34(Suppl.), S117–S120. doi: 10.1002/eat.10211

Submitted: January 22, 2010 Revised: August 6, 2010 Accepted: August 8, 2010

A Tutorial on Count Regression and Zero-Altered Count Models for Longitudinal Substance Use Data

Acknowledgement: We thank Eun-Young Mun, Zac Imel, Isaac Rhew, Jennifer Kirk, and an anonymous reviewer for helpful feedback that improved the manuscript in numerous ways. The example data included in the article were collected via the National Institute on Alcohol Abuse and Alcoholism (NIAAA) Grants AA016099 and AA014576 (Clayton Neighbors, PI), and David C. Atkins' time was supported in part by NIAAA Grant AA019511 (Eun-Young Mun, PI).

What is the impact of personalized normative feedback on drinking related problems in college students over time? How does weekend versus weekday drinking vary by gender and fraternity/sorority status when assessed on a daily basis? Many questions about alcohol and substance use focus on change across time, and the methods used to analyze these questions need to account for the longitudinal nature of the data. Generalized linear mixed models (GLMMs; Gelman & Hill, 2007; Hedeker & Gibbons, 2006; also called hierarchical [or multilevel] generalized linear modeling, Raudenbush & Bryk, 2002; Snijders & Bosker, 1999) are increasingly common analytic approaches for longitudinal data, given their flexible handling of unbalanced repeated measures (i.e., individual participants may have unique numbers and timings of assessments) and the widespread availability of software for estimating such models. Moreover, GLMMs are appropriate for continuous as well as discrete outcomes.

However, the distributions of alcohol and substance abuse outcomes have characteristic shapes: They are often positively skewed and bounded by zero. Moreover, there can be a large stack of data points at zero, indicating individuals and/or occasions without drinking, use, or related problems. These distributions reflect that alcohol and substance use outcomes are often count data, representing a total number of something, be it drinks, days using, or number of problems. Except in special circumstances (e.g., specially selected samples with high drinking or drug use), statistical models that assume normally distributed residuals will provide poor fit to such data and will lead to incorrect confidence intervals and p values. Instead, count regression approaches such as Poisson or negative binomial regression or zero-altered count models (e.g., zero-inflated or hurdle models) are much more appropriate for these types of data (Atkins & Gallop, 2007; Coxe, West, & Aiken, 2009; Hilbe, 2011; Neal & Simons, 2007; Simons, Neal, & Gaher, 2006).

In the past, addictions researchers have often ignored (or not been aware of) violations of distributional assumptions or have attempted to deal with them in nonoptimal ways. Count regression models are beginning to be applied to addictions data (e.g., Gaher & Simons, 2007; Lewis et al., 2010), but accessible resources on how to apply these models to longitudinal data are scarce. The present article provides a tutorial in analytic methods for count data from longitudinal studies, focusing on extensions to GLMMs for count outcomes. We use two examples from our research to illustrate the need for, and application of, longitudinal count models. Data and computer code to run the analyses in R, Mplus, SAS, Stata, and SPSS are available on a supplementary website (http://depts.washington.edu/cshrb/newweb/statstutorials.html), though note that not all software can run all models that are covered here at present time. The outline of the article is as follows: Introduction to example data and research questions, brief overview of count regression models, GLMMs for count regression models, analyses and interpretation of example data, and discussion of software and practical issues in using these methods. In addition, there is a technical appendix containing important, but more advanced, material (see online supplemental material; http://dx.doi.org/10.1037/a0029508.supp). We assume that readers have a basic familiarity with linear mixed models (i.e., hierarchical linear or multilevel models assuming normally distributed errors) and count regression models, though both are introduced briefly here and introductory resources are highlighted throughout.

The first example dataset is drawn from an intervention study aimed at reducing problematic drinking in college students (Neighbors et al., 2010). The current article focuses on gender differences across 2 years in alcohol-related problems, as measured by the Rutgers Alcohol Problem Index (RAPI; White & Labouvie, 1989). The dataset includes 3,616 repeated measures across five time points from 818 individuals. The second dataset involves intensive, daily assessment of drinking. The data come from a larger intervention study of event-specific prevention (i.e., drinking related interventions for 21st birthdays and spring break), but the current data are observational. Specifically, these data record the number of drinks for each day over approximately the last 30 days for 980 individuals (23,992 total person-days), as measured by the timeline follow-back interview (TLFB). These data came from a survey study of 21st birthday drinking and consequently include some extreme drinking events relative to a random sample of student's drinking (Neighbors et al., 2011). Analyses focus on drinking differences by gender, Greek status (i.e., fraternity/sorority membership), and weekend (Thurs–Sat) versus weekday (Sun–Wed). (Note that Thursday was included as part of the “weekend” given that drinking on Thursday was more similar to Friday and Saturday drinking than other days of the week.)

Count variables are often positively skewed and often include many observations at zero. The top row of Figure 1 displays (unconditional) distributions of drinking (TLFB) and alcohol problems (RAPI), which are strongly skewed with a mode of zero. The bottom row of Figure 1 shows histograms of residuals from regressing the RAPI on gender and time assuming normally distributed residuals (i.e., ordinary least squares [OLS] regression): on the left without any transformation and on the right with a log transformation of the RAPI. These plots show that skewed count regression outcomes will rarely meet the distributional assumptions of OLS regression or linear mixed models. Moreover, count outcomes will also typically violate the equal variances (i.e., homoskedasticity) assumption of linear models as count outcomes have a direct relationship between their mean and variance, where higher levels of the outcome have greater variance. Although transforming the outcome is a commonly suggested strategy for skewed data, a stack of zeroes will not be smoothed out by a transformation. Moreover, focusing on the TLFB data, the distribution shown in Figure 1 is suggestive of two different types of associations or research questions: (a) what is related to no drinking versus any drinking (i.e., zero vs. nonzero), and (b) what is related to the amount of drinking when there is drinking? Although the zero/nonzero aspects of the RAPI data are not quite as notable, a similar set of questions could be asked of the RAPI data. As will be described later, zero-inflated and hurdle models have submodels that focus on these two questions. Prior to introducing count regression models, we consider the qualities of the example data that are related to the questions just posed.

Figure 1. Plots of frequency counts of daily drinks from timeline follow-back (TLFB; upper left) and Rutgers Alcohol Problems Index (RAPI; upper right). Residuals from fitting an ordinary least squares regression to the RAPI or log-transformed RAPI are in lower left and lower right, respectively.

Consider how the proportion of individuals drinking versus not drinking and number of drinks on drinking days are related to covariates in the TLFB data. Figure 2 presents means and 95% confidence intervals (CIs) for number of drinks on drinking days (top half of graph) and proportion of days drinking (bottom half of graph) by weekday versus weekend, Greek status, and gender.

Figure 2. Means and 95% confidence intervals for drinking on drinking days (top row) and proportion of individuals drinking (bottom row). Means are stratified by weekend versus weekday, male versus female, and fraternity/sorority member or not.

There are strong differences in the proportion of people drinking for weekend versus weekday, with less prominent differences for number of drinks when drinking. Moreover, from this descriptive view, the number of drinks is highly related to fraternity/sorority status, whereas proportion of people drinking does not appear as strongly related to Greek status.

Figure 3 shows a similar set of plots for the RAPI by gender across time. As with the TLFB drinking data, patterns of association appear different across the two aspects of the outcome. Over the five assessments (and following intervention for most participants), the proportion of individuals with any alcohol problems is dropping, with a greater proportion of men consistently reporting more alcohol problems (right panel). When examining the number of alcohol problems (given some problems), we see evidence for divergence between the sexes, with women showing slight decreases in number of alcohol problems, whereas men appear to show slight increases, though with notable variability as seen in the confidence intervals (left panel). Thus, these graphs serve to highlight that different associations can occur with outcomes with notable zeroes (i.e., whether there is any drinking [zero vs. not zero] and amount of drinking when any drinking).

Figure 3. Means and 95% confidence intervals for number of problems when there are any problems (left) and proportion of individuals reporting any drinking-related problems (right). Means are stratified by assessment period and gender.

Before moving on to discuss GLMMs for count outcomes, we briefly introduce count regression models (more thorough introductions can be found in articles by Atkins & Gallop, 2007, and Coxe et al., 2009, and the book by Hilbe, 2011). The basic count regression model is Poisson regression, which is one of the generalized linear models (McCullagh & Nelder, 1989). There are two critical differences between OLS regression and Poisson regression. First, the outcome (conditional on covariates) is assumed to be distributed as a Poisson random variable as opposed to a Normal random variable. The top row of Figure 4 depicts three different Poisson distributions, with varying means (denoted by the Greek letter mu). This figure underscores that the Poisson distribution is a discrete distribution for non-negative integers, the exact qualities of count variables (i.e., a count variable cannot be negative or fractional). Second, in Poisson regression the linear predictor of the regression model (i.e., the right-hand side of the regression equation) is connected to the outcome via a natural logarithm link function. Although this is not identical to transforming the outcome, it does mean that the regression coefficients from a Poisson model are on a log scale. Similar to logistic regression, raw coefficients are typically raised to the base e (i.e., exponentiated, the antilog function) and interpreted as rate ratios. Like odds ratios (ORs), rate ratios are inversely proportional around one (i.e., a rate ratio of 3 is equal in strength but opposite in direction to a rate ratio of 1/3). Rate ratio interpretation will be described in greater detail later, in the applications section.

Figure 4. Plots of frequency counts of data simulated from Poisson distributions with three different means (top row). The bottom row contains frequency counts of data simulated from negative binomial distributions with the same mean but varying dispersion.

The Poisson distribution (and regression) has an Achilles' heel of sorts in that it has the property that the mean equals the variance. In real data the variance often far exceeds the mean, and we would say that the data are overdispersed relative to the Poisson distribution. Using descriptive statistics, dispersion is typically defined by the ratio of the variance to the mean. Thus, a Poisson distribution assumes a dispersion parameter of 1 (also called equidispersion). A dispersion parameter of 3 would indicate a variance value of three times the mean, which would (descriptively) indicate overdispersion. Both of our example datasets show evidence of overdispersion simply based on descriptive statistics (TLFB: M = 1.2, Var = 9.3; RAPI: M = 6.3, Var = 82.9), though overdispersion can also be influenced by the longitudinal nature of these data and may be accounted for by covariates included in the model. Note that this description is meant to convey the intuitive ideas underlying overdispersion, but formal tests can (and should) be used in regression modeling of count data (see, e.g., Hilbe, 2011).

The negative binomial model extends the Poisson model by allowing the mean and variance to be different. The lower row of Figure 4 presents three different negative binomial distributions. Note that each of these distributions has the same mean, but the dispersion varies, highlighting the primary difference between the Poisson and negative binomial distributions. The Poisson regression model is a special case of the negative binomial model, and when the mean equals the variance, the two will yield identical results. However, when the variance exceeds the mean, the negative binomial is more appropriate, and its standard errors will be reliably larger than those from the Poisson, reflecting the additional variance in the outcome. In practice, the Poisson model is rarely a good fitting model for exactly this reason, and selecting a Poisson model when the data are overdispersed will yield overly liberal statistical tests (i.e., p values will appear significant when they are not in reality, using a more appropriate model).

As seen in Figure 4, the negative binomial regression model can fit highly skewed data, including data with a relatively large number of zeroes. However, when there is a clear stack of zeroes in the data and especially when the nonzero distribution is not a smooth extension from the zeroes, alternative models may be appropriate. Two closely related models explicitly handle count data with zeroes above and beyond what would be predicted by a negative binomial model: zero-inflated models and hurdle models (see, e.g., Hilbe, 2011; Zeileis, Kleiber, & Jackman, 2008). Both models include a logistic regression for the zeroes in the data and a count regression (either Poisson or negative binomial) for the counts. However, zero-inflated and hurdle models take different approaches to dividing the data around zero. The predicted zeroes in a zero-inflated model come from both the count distribution and an extra mass of zeroes and are a type of mixture model in which the distribution at zero arises from two sources (i.e., count and logistic submodels). Thus, the logistic regression model in a zero-inflated model is for “excess zeroes,” over and above what would be predicted by the count distribution. Hurdle models, on the other hand, model all the zeroes in the logistic regression, and nonzero counts are modeled by a truncated count regression (i.e., truncated because it does not include zero). Hurdle models are related to the more general class of models, often called “two-part models,” in which a logistic model for zero versus nonzero is combined with a model for nonzero values. In the current article we consider only Poisson and negative binomial for nonzero models, but this does not have to be the case (see, e.g., health care costs as in Buntin & Zaslavsky, 2004). In many instances zero-inflated models and hurdle models will yield similar results; however, hurdle models are more straightforward to interpret as all zeroes are handled in one portion of the model, and computationally, hurdle models are somewhat easier to fit as the two parts of the model can be fit independently of one another.

All of the count regression models introduced thus far assume independent observations. This assumption is violated in longitudinal data because repeated observations on the same individual will be correlated. To accommodate this nonindependence of observations, mixed model extensions to count regressions can be used.

The top half of Figure 5 displays change across time in the RAPI for eight randomly selected individuals. As seen in that plot individuals started the study with a wide range of alcohol problems, and some individuals made notable changes in alcohol problems, whereas others did not. To illustrate the variability in intercepts (i.e., initial RAPI) and slopes (i.e., change over time in RAPI), Poisson regressions were fit to each individual's data (i.e., RAPI was regressed on time for each individual separately, using a Poisson regression; see Singer & Willett, 2003, for general discussion of this approach). The distributions of intercepts and slopes are plotted in the bottom of Figure 5. The means of these distributions are descriptive estimates of average intercepts (M = 8.2) and slopes (M = 0.99), and the distributions themselves highlight the variability across individuals in intercepts and slopes.

Figure 5. RAPI scores plotted over time for eight randomly selected individuals (top row). Distribution of intercepts (bottom left) and slopes (bottom right) from fitting separate Poisson regressions to each individual's data.

GLMMs for longitudinal data extend this logic of individual growth curves by including subject-specific variability in one or more components of the model (e.g., intercepts, slopes) via additional variance terms that describe a distribution of regression coefficients across individuals within the study. GLMMs have been referred to as random coefficient models for precisely this reason. Using the systems of equations or hierarchical format with explicit Level 1 and Level 2 models, a Poisson GLMM for the RAPI data might be       which is identical to the composite form of the model by substituting the Level 2 equations into their Level 1 counterparts:   where t indexes time, i indexes individuals, and the linear predictor (i.e., right-hand side of equation) is connected to the mean of the outcome via a natural logarithm link function. Male is a dummy-variable for gender (female = 0, male = 1), and Time measures time in months since the start of the study. The two variance terms (u0i, u1i) describe the deviations of each participant's intercept and slope around the overall trajectory for the sample, defined by the fixed effects (i.e., b0+ b1Male + b2Time). As is common with error terms, these variances are assumed (multivariate) normally distributed. Finally, the model assumes that the conditional distribution of the outcome given fixed and random effects is Poisson distributed. It is important to note that both the fixed and random effects are connected to the outcome via the link function, which has implications for interpretation of the model (see discussion of marginal vs. conditional effects, below).

The results for the model in Equation 1 are presented in Table 1. Focusing first on the fixed effects, the subject-specific rate ratios (RRs) are simply the exponentiated coefficients (e.g., for the intercept, e^1.45 = 4.28). The intercept RR provides the estimated alcohol problems at baseline for women (i.e., when all covariates are equal to zero, as in all regression models), conditional on the random effects. The RR of 1.29 for men indicates that their alcohol problems are 29% higher than women on average. Generally, the distance above or below 1 is interpreted as the percentage increase or decrease in the outcome for a one-unit increase in the predictor. Similarly, the 0.97 RR for time implies a 3% reduction in RAPI for each 1-month increase. Let's consider the interpretation of RRs a bit more closely. The predicted regression lines based on the fixed effects are shown in Figure 6, and the specific values are presented in the figure as well. If this were a linear mixed model or OLS regression, the predicted regression lines would be perfectly straight and parallel (i.e., because there is no interaction between Male and Time), but it is clear from the figure that the predicted regression lines for men and women from the Poisson GLMM are curved and are getting closer to one another over time. Poisson models with their log link functions are sometimes called multiplicative models, whereas OLS regression (and linear mixed models) are considered additive. For the latter, the regression coefficients tell us how much to “add” to our prediction for each one-unit increase in a covariate.

GLMM Poisson Results for RAPI Data

Figure 6. Predicted (unit-specific) RAPI scores from Poisson GLMM for men and women over time, with specific values noted in text.

The interpretation of multiplicative models is more complicated. First, we exponentiate the fixed effects in Equation 1b:   where exp is the exponentiating function, raising to the base e. We then reexpress Equation 1c using a property of exponetiated sums:  

Equation 1d shows why Poisson models are considered multiplicative: The exponentiated coefficients—the RRs—provide a multiplying factor for each unit change in the covariate. Thus, the RR of 1.29 for men indicates that men's predicted alcohol problems are 1.29 times women's predicted alcohol problems at each level of time, which is identical to saying that it is 29% higher (e.g., at time = 0, 4.28 × 1.29 = 5.52, and at time = 24 months, 2.03 × 1.29 = 2.62). For every 6 months, the model predicts alcohol problems are falling by e^−0.03∗6 = 0.83 (i.e., an RR for each 6-month change in time). We can similarly confirm that each 6-month change in predicted alcohol problems is 17% less (i.e., multiply by 0.83) than the preceding predicted value. Thus, for OLS regression and linear mixed models, coefficients describe the amount added to predictions with each one-unit covariate change, whereas Poisson and Poisson GLMM coefficients describe the amount multiplied.

One critical feature of these coefficients that has not been widely noted in the applied literature is that these coefficients (and their interpretation) are conditional on specific values of the random-effects distribution. In linear mixed models, fixed-effect coefficients can be interpreted as “averaging over” the random effects (sometimes called marginal coefficients), but this is not true in GLMM with nonidentity link functions (see, e.g., Heagerty & Zeger, 2000 and Raudenbush & Bryk, 2002, Chapter 10). This issue of conditional versus marginal coefficients (also called unit-specific vs. population-average) is discussed in the online supplemental material (see http://dx.doi.org/10.1037/a0029508.supp), along with additional resources.

There are several considerations in judging how well the model fits the data. First, similar to all regression models, we should consider whether we have the correct predictors included in the model and whether any terms have nonlinear associations with the outcome. These decisions should be informed by past research, theory, and thorough descriptive statistics and graphs. In fact, it is hard to overemphasize the importance of thorough descriptive analyses to help guide decisions in building complicated models, such as those considered here. In addition, similar to many regression models, GLMMs assume that variance terms are normally distributed and homoskedastic, and plots of these assumptions can be found in the accompanying online materials (see eFigure 1 in online supplementary material at http://dx.doi.org/10.1037/a0029508.supp). Second, similar to linear mixed models we should consider which terms in our model should have corresponding random effects; with GLMMs decisions about which random effects to include are usually determined by testing nested models via deviance tests (also called likelihood ratio chi-square tests; Molenberghs & Verbeke, 2005). Based on descriptive statistics and graphs as well as deviance tests for random effects, the initial model for the RAPI was extended to include an interaction between gender and time and an additional per-observation random effect (discussed below). These results are shown under Model 2 in Table 1.

Most (but not all) GLMMs are fit via a procedure called maximum likelihood, and when the iterative fitting procedure stops, a deviance statistic is reported (technically, −2 × log-likelihood, and sometimes reported as −2LL). The difference between deviances from two nested models is distributed as a chi-square random variate and hence can be tested using a chi-square distribution with degrees of freedom equal to the difference in parameters between the models (see Singer & Willett, 2003, for further details). Deviance tests can also be used with count regression GLMMs, with one major caveat. Fitting GLMMs with non-normal outcomes (e.g., binary or count outcomes) is considerably more challenging, and there are a variety of optimization strategies and algorithms. Only certain algorithms have been shown to be accurate enough to allow deviance tests of nested models. A brief overview of algorithms and approaches is presented in the online supplemental material (see http://dx.doi.org/10.1037/a0029508.supp).

There are two final, critical aspects of model evaluation for GLMMs for count outcomes: overdispersion and zeroes. As noted earlier, the Poisson GLMM assumes that, conditional on fixed and random effects, the distribution of the outcome is distributed as a Poisson variable, with its property of equidispersion (i.e., mean equal to the variance). The most straightforward way to examine whether data are overdispersed relative to the Poisson is to fit a model that allows for overdispersed data. Logically, we might consider the negative binomial. For technical details that go beyond our focus here, the negative binomial model is challenging (though not impossible) to extend to random effects (see, e.g., Hilbe, 2011, pp. 488–501), though some software packages are beginning to incorporate the negative binomial with random effects (e.g., Mplus, R, SAS). With GLMMs we can extend the Poisson model to include a per-observation error term, which captures overdispersion. This type of model is often called an overdispersed Poisson model, which is functionally similar to a negative binomial model (see Rabe-Hesketh & Skrondal, 2008, pp. 389–390, or Ver Hoef & Boveng, 2007, for further discussion). Moreover, adding the extra error term is very similar to a residual error term, as in models for normally distributed data, and can be assessed via a deviance test. Model 2 in Table 1 fits an overdispersed Poisson GLMM, incorporating this extra error term. A deviance test for assessing the improvement in fit due to the addition of the per-observation error term is highly significant, χ²(1) = 2,032.8, p < .01.

Earlier, we commented on the number of zeroes descriptively in the RAPI data, and a final component with count regression models is to examine how well the model replicates the number of zeroes, and more generally, the distribution of counts in the raw data; that is, what does the model predict the histogram of counts looks like, and specifically, what the zeroes look like? A figure included with the supplementary online material presents the raw distribution of the RAPI along with estimates from model two in Table 1 (see Figure 2 at http://dx.doi.org/10.1037/a0029508.supp). Although Model 2 appears quite accurate for counts greater than 2, there is noticeable lack of fit for counts of 0, 1, and 2. Essentially, the Poisson distribution is not flexible enough to fit the steep drop of counts from 0 to 1, with much more moderate change for counts greater than 1. As a result the raw RAPI data has 756 zeroes and 395 ones, whereas the final overdispersed Poisson GLMM predicts 616 zeroes and 546 ones. This is a primary statistical motivation for considering a zero-inflated or hurdle mixed model.

Following our example with the RAPI, a hurdle mixed model would be     where t indexes time, i indexes individuals, l and c index logistic and count portions of the model, and p is the proportion of RAPI scores greater than 0. As noted earlier, zero-inflated and hurdle models have two submodels, one related to the zeroes and a second related to the counts. The key difference between hurdle and zero-inflated models is how they handle zeroes: Hurdle models cleanly divide the models, with all zeroes accounted for in the logistic regression, whereas zero-inflated models treat the zeroes as a mixture (i.e., both submodels in zero-inflated models contribute zeroes). As we saw with the overdispersed Poisson GLMM, the hurdle GLMM presented above adds random effects to the linear predictors, with the major difference now being that there are two linear predictors. The random intercept in the logistic model implies that there is variability across individuals in the likelihood of reporting any problems, whereas the random intercept and slope in the count regression models variability in the intercept and change across time in number of problems when there are problems reported. A random slope for time in the logistic portion would model individual variability over time in proportion of zeroes.

Prior to fitting the hurdle mixed model, let us say a word about when to use zero-inflated or hurdle models. Similar to many model selection decisions, the choice between models should include statistical considerations, theoretical considerations, and parsimony. As noted earlier the overdispersed Poisson GLMM of the RAPI data underpredicted zeroes (756 in the raw data vs. 616 predicted by the model), and with the TLFB data, the same type of model led to a highly non-normal distribution for the per-observation error term, based on a histogram and quantile-quantile plot of this error term. Thus, in both of these cases a zero-inflated or hurdle mixed model might be preferred on statistical grounds. However, the statistical motivations just noted do not automatically mean that covariates have important and distinct relationships across the logistic and count portions of a hurdle (or zero-inflated) model. That is, sometimes a hurdle or zero-inflated mixed model will fit the data better, but the conclusions are largely the same compared to a Poisson GLMM (typically because the covariates are primarily related to the nonzero counts). In this scenario, parsimony might suggest the simpler Poisson GLMM is adequate as it yields similar conclusions, though we strongly suggest fitting the more complex model to test whether the simpler model is adequate. Finally, theory may make clear predictions about the proportion of days with any drinking or the amount drunk on days with drinking. These would clearly prefer hurdle or zero-inflated models.

Results from the hurdle mixed model shown in Equations 2a and 2b are presented in Table 2. In examining Table 2, we find that these results map on to the earlier descriptive graphs quite closely. In particular, the count portion of the model shows that men tend to have more alcohol problems when there are problems at the start of the study (RR = 1.21). Because of the coding of the gender dummy-variable, the main effect of time describes the change in women's alcohol problems across time. The RR of 0.98 implies that, conditional on the random effects, women's RAPI (i.e., alcohol problems when there are problems) is dropping 2% with each successive assessment, whereas the interaction between gender and time implies that men's alcohol problems (when there are problems) are barely changing at all, which is confirmed by estimating the men's simple slope for time, RR = 0.99, 95% CI [0.98, 1.00].

Over-Dispersed Poisson Hurdle Mixed Model Results for RAPI Data

The logistic portion of the model describes the proportion of the sample reporting some alcohol problems. (Note that software for zero-inflated and hurdle models are not always consistent as to whether the logistic portion predicts zeroes as opposed to nonzeroes. Here it is predicting nonzeroes.) The ORs in Table 2 show that there are no differences between the sexes in the proportion of individuals reporting alcohol problems at the start of the study, OR = 0.83, 95% CI [0.59, 1.18]. For women, the odds of reporting any alcohol problems decreases by 3% per month, OR = 0.97, 95% CI [0.95, 0.99], conditional on the random effects. For men, their rate of change is somewhat slower, OR = 1.02, 95% CI [0.99, 1.04].

In turning to the TLFB data, the earlier plots showed that there was a very large stack of zeroes and that covariate effects (i.e., gender, weekend, and fraternity/sorority status) might vary across submodels defined by likelihood of any drinking (i.e., zero vs. not zero) and amount of drinking when there is drinking. These considerations would point toward a zero-inflated or hurdle model; moreover, preliminary models using an overdispersed Poisson GLMM showed that there was a very poor fit between predicted and observed count distribution, and the per-observation random-effect distribution was not close to being normally distributed. Given these considerations, a hurdle overdispersed Poisson GLMM was fit to the TLFB data, fitting main and two-way interaction effects in both submodels via three dummy variables: Male (0 = Women, 1 = Men), Weekend (0 = Sun/Wed, 1 = Thurs/Sat), and FratSor (0 = Not in fraternity/sorority, 1 = Fraternity/Sorority). A random intercept and random slope for weekend were included in both the logistic and truncated count submodels, as well as the per-observation random intercept in the truncated count submodel. Conditional (or unit-specific) fixed-effects are shown in Table 3. In the logistic portion of the model, there is a single, significant effect for weekend: Many more students drink, regardless the quantity, on the weekends as compared to the weekdays, OR = 3.06, 95% CI [2.60, 3.54]. In the count submodel, all three two-way interactions are significant. One approach to interpreting interactions is to use “simple slopes” (see, e.g., Jaccard, 2001). However, in the present case, the model is estimating mean drinking on drinking days for eight specific groups—for each combination of gender, weekend, and fraternity/sorority status. To help interpret the findings of the count submodel, we estimated marginal (or population-average), predicted means for each of these eight groups, which are shown in Figure 7. The figure reveals several interesting findings. Not surprisingly, fraternity and sorority members drink more on drinking days than students who are not in fraternities and sororities, and fraternity and sorority members' drinking is similar on weekdays versus weekend—when they are drinking (i.e., the count submodel in a hurdle is only for nonzeroes). However, contrasts of weekend versus weekday drinking for men and women who are not in fraternities and sororities shows that they do drink more on weekends, women: RR = 1.07, 95% CI [0.99, 1.14], p = .06; men: RR = 1.20, 95% CI [1.12, 1.28], p < .01. Thus, the hurdle mixed model provides a more nuanced view of drinking differences across these groups, parsing out drinking versus not drinking and mean drinking on drinking days.

Over-Dispersed Poisson Hurdle Mixed Model Results for TLFB Data

Figure 7. Predicted (marginal) drinks for each unique combination of gender, weekend (WE) versus. weekday (WD), and fraternity/sorority status from count submodel of overdispersed Poisson hurdle mixed model.

Statistical software packages are steadily expanding their coverage of GLMMs and the models covered in the current paper. Table 4 presents an overview of SPSS, Stata, R, SAS, and Mplus, describing which of the models covered in the present article are possible to fit. As seen there, the basic Poisson GLMM is implemented in all software, whereas negative binomial GLMM and zero-altered models are less widely available. The online materials provide computer code for fitting some example models in each of the packages mentioned, and the online supplementary material covers additional details related to fitting GLMM, which is both more technical but still practically important (see http://dx.doi.org/10.1037/a0029508.supp). Not all software packages were able to fit all models described in the present article (e.g., SAS NLMIXED can fit zero-inflated mixed models but could not fit all the variance components describe in the TLFB example). Over the coming years, there will likely be greater software availability for fitting the types of models covered here as well as more robust fitting algorithms.

Comparison of Software for Fitting Generalized Linear Mixed Models (GLMM)

Summary

The present article discussed extensions to count regression and zero-altered count regression models to longitudinal data based on GLMM. We hope that this presentation, along with the appendix and available data and code, helps addiction researchers to learn and appropriately apply these models. Given the research designs and data that addiction researchers often collect, the models covered here are often the most appropriate analytic tools. At the same time, like other statistical models that are finding their way to the applied research literature (e.g., growth mixture models; Muthén & Muthén, 2000), multilevel count models are complex. The types of assumptions and general considerations in model fitting grow exponentially from linear models to generalized linear models to linear mixed models to generalized linear mixed models. We encourage researchers to explore and use these tools as their hypotheses and data merit, though also note that they require a serious commitment of time and study to be used appropriately.

¹ The six graphs in Figure 4 were created by simulating 1,000 random draws from the specified Poisson or negative binomial distributions. Thus, these are similar to frequency histograms, except there is no binning of multiple values.

² There are alternatives to the negative binomial distribution in terms of dealing with overdispersed count data, including quasi-Poisson models, Poisson-normal models, and robust standard errors. Details on these models can be found in Hilbe (2011).

³ Note that the distribution of slopes appears somewhat odd, with spikes at extremely high or low values. This reflects that some individuals have incomplete data, and hence, slopes fit to two or three points of data can yield very extreme values. The subject-specific predictions from GLMMs pool information about the sample and the individual's data, leading to more sensible fits for individuals (see, e.g., discussion of empirical Bayes estimates in Raudenbush & Bryk, 2002).

⁴ The notation in Equation 1 is slightly nonstandard. In the statistical literature it is common to see GLMMs described in terms such as: η = XB + Zb, with g(μ) = η, and g( ) = log. Translating into words this means that η is the linear predictor, or right-hand side of the model, including both fixed and random effects (i.e., XB and Zb, respectively). The linear predictor is connected to the mean of the outcome (i.e., μ) via some function. With the Poisson model, the link function is the log.

⁵ Estimating the predicted number of zeroes based on a GLMM takes us into somewhat more technical material. The Poisson GLMM can be represented by Pois(XB + Zb), where X is the design matrix of the fixed-effects; B is a vector of fixed-effects coefficients; Z is the design matrix of the random effects; and b is a vector of random-effects variances. The key thing to realize is that the formula within the parentheses defines the fitted values of the model. Thus, we can read this equation as follows: The data are distributed as a Poisson random variable with a mean structure defined by the fixed and random effects of the GLMM. Similar to the discussion of conditional effects, the random effects create additional challenges in estimating the model-based distribution of counts. The accompanying R code shows one method for simulating from the fitted model to estimate the predicted distribution of counts.

⁶ We note in passing that the hurdle mixed models reported in this article were fit using the MCMCglmm package (Hadfield, 2010) in R, which uses a Bayesian approach to model estimation. For many practical problems maximum likelihood estimation and Bayesian MCMC estimation will yield similar if not identical results, though there are basic differences in both inferential philosophy as well as estimation. A brief discussion is provided in the technical appendix of the online material (see http://dx.doi.org/10.1037/a0029508.supp). In this particular instance, the choice is quite pragmatic as MCMCglmm provides the most flexible option for fitting these types of models in R.

⁷ To estimate marginal (or population-average) estimates from a GLMM, it is necessary to include both fixed and random effects. The online supplementary material provides details and the example of how the present estimates were created (see http://dx.doi.org/10.1037/a0029508.supp).

Atkins, D. C., & Gallop, R. J. (2007). Rethinking how family researchers model infrequent outcomes: A tutorial on count regression and zero-inflated models. Journal of Family Psychology, 21, 726–735. doi:10.1037/0893-3200.21.4.726

Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J. S. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology and Evolution, 24, 127–135. doi:10.1016/j.tree.2008.10.008

Breslow, N. E., & Clayton, D. G. (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88, 9–25. doi:10.2307/2290687

Buntin, M. B., & Zaslavsky, A. M. (2004). Too much ado about two-part models and transformation? Comparing methods of modeling Medicare expenditures. Journal of Health Economics, 23, 525–542. doi:10.1016/j.jhealeco.2003.10.005

Coxe, S., West, S. G., & Aiken, L. S. (2009). The analysis of count data: A gentle introduction to Poisson regression and its alternatives. Journal of Personality Assessment, 91, 121–136. doi:10.1080/00223890802634175

Draper, D. (2008). Bayesian multilevel analysis and MCMC. In J.de Leeuw and E.Meijer (Eds.), Handbook of multilevel analysis (pp. 77–139). New York, NY: Springer. doi:10.1007/978-0-387-73186-5_2

Gaher, R. M., & Simons, J. S. (2007). Evaluations and expectancies of alcohol and marijuana problems among college students. Psychology of Addictive Behaviors, 21, 545–554. doi:10.1037/0893-164X.21.4.545

Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. New York, NY: Cambridge University Press.

Hadfield, J. D. (2010). MCMC methods for multi-response generalized linear mixed models: The MCMCglmm R package. Journal of Statistical Software, 33, 1–22.

Heagerty, P. J., & Zeger, S. L. (2000). Marginalized multilevel models and likelihood inference. Statistical Sciences, 15, 1–26. doi:10.1214/ss/1009212670

Hedeker, D., & Gibbons, R. D. (2006). Longitudinal data analysis. Hoboken, NJ: John Wiley & Sons.

Hilbe, J. M. (2011). Negative binomial regression (2nd ed.). New York, NY: Cambridge University Press. doi:10.1017/CBO9780511973420

Jaccard, J. (2001). Interaction effects in logistic regression. Newbury Park, CA: Sage.

Lewis, M. A., Neighbors, C., Geisner, I. M., Lee, C. M., Kilmer, J. R., & Atkins, D. C. (2010). Examining the associations among severity of injunctive drinking norms, alcohol consumption, and alcohol-related negative consequences: The moderating roles of alcohol consumption and identity. Psychology of Addictive Behaviors, 24, 177–189. doi:10.1037/a0018302

Liang, K. Y., & Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13–22. doi:10.1093/biomet/73.1.13

Lynch, S. M. (2007). Introduction to applied Bayesian statistics and estimation for social scientists. New York, NY: Springer. doi:10.1007/978-0-387-71265-9

McCullagh, P., & Nelder, J. A. (1989). Generalized linear models (2nd ed.). Boca Raton, FL: Cahpman & Hall/CRC.

Molenberghs, G., & Verbeke, G. (2005). Models for discrete longitudinal data. New York, NY: Springer.

Muthén, B., & Muthén, L. (2000). Integrating person-centered and variable-centered analysis: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical and Experimental Research, 24, 882–891. doi:0145-6008/00/2406-0882

Neal, D. J., & Simons, J. S. (2007). Inference in regression models of heavily skewed alcohol use data: A comparison of ordinary least squares, generalized linear models, and bootstrap resampling. Psychology of Addictive Behaviors, 21, 441–452. doi:10.1037/0893-164X.21.4.441

Neighbors, C., Atkins, D. C., Lewis, M. A., Lee, C. M., Kaysen, D., Mittmann, A., . . .Rodriguez, L. M. (2011). Event specific drinking among college students. Psychology of Addictive Behaviors, 25, 702–707. doi:10.1037/a0024051

Neighbors, C., Lewis, M. A., Atkins, D. C., Jensen, M. M., Walter, T., Fossos, N., . . .Larrimer, M. E. (2010). Efficacy of web-based personalized normative feedback: A two-year randomized controlled trial. Journal of Consulting and Clinical Psychology, 78, 898–911. doi:10.1037/a0020766

Rabe-Hesketh, S., & Skrondal, A. (2008). Multilevel and longitudinal modeling using Stata. College Station, TX: Stata Press.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications.

Raudenbush, S. W. (2008). Many small groups. In J.de Leeuw & E.Meijer (Eds.), Handbook of multilevel analysis (pp. 207–236). New York, NY: Springer.

Rodriguez, G., & Goldman, N. (2001). Improved estimation procedures for multilevel models with binary response: A case-study. Journal of the Royal Statistics Society, Series A, 164, 339–355.

Simons, J. S., Neal, D. J., & Gaher, R. M. (2006). Risk for marijuana-related problems among college students: An application of zero-inflated negative binomial regression. American Journal of Drug and Alcohol Abuse, 32, 41–53. doi:10.1080/00952990500328539

Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York, NY: Oxford University Press.

Snijders, T. A. B., & Bosker, R. J. (1999). An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage Publications.

Ver Hoef, J. M., & Boveng, P. L. (2007). Quasi-Poisson vs. negative binomial regression: How should we model overdispersed count data?Ecology, 88, 2766–2772. doi:10.1890/07-0043.1

White, H. R., & Labouvie, E. W. (1989). Towards the assessment of adolescent problem drinking. Journal of Studies on Alcohol, 50, 30–37.

Zeileis, A., Kleiber, C., & Jackman, S. (2008). Regression models for count data in R. Journal of Statistical Software, 27, 1–25.

Submitted: November 22, 2011 Revised: June 15, 2012 Accepted: June 18, 2012

Correction to Atkins et al. (2012)

In the article “A Tutorial on Count Regression and Zero-Altered Count Models for Longitudinal Substance Use Data” by David C. Atkins, Scott A. Baldwin, Cheng Zheng, Robert J. Gallop, and Clayton Neighbors (Psychology of Addictive Behaviors, Vol. 27, No. 1, pp. 166–177. doi:10.1037/a0029508), the URL for the supplemental material was incorrect throughout the text due to a production error.

Supplemental material for this article is available at: http://dx.doi.org/10.1037/a0029508.supp. The online version of this article has been corrected.

Adolescent Change Language Within a Brief Motivational Intervention and Substance Use Outcomes

Acknowledgement: The research reported herein was supported by National Institute on Drug Abuse Grant R01 DA15751. We gratefully acknowledge the contributions of research assistants Sarah Bowen and Dana Rhule; counselors Jennifer Mullane, Melissa Phares, and Maija Ryan; data coders Avry Todd and Kate Hallman; the staff at New Horizons Ministries; and Dan Kivlahan for comments on a draft of this article.

Motivational interviewing (MI; Miller & Rollnick, 2002) is a popular, empirically based counseling method for a range of health-related problems. Defined as a “directive, client-centered counseling style for eliciting behaviour change by helping clients explore and resolve ambivalence” (Rollnick & Miller, 1995, p. 326), MI emphasizes formation of collaborative therapeutic relationships with clients through which client language about change may be strategically elicited and reinforced. Change language is defined as client expressions of problems with the current state, benefits of change, and hope and optimism about future change (Miller & Rollnick, 2002). In articulating a theory for MI, Hettema, Steele, and Miller (2005) stated that (a) the practice of MI should elicit increased levels of change and decreased levels of resistance from clients, (b) the extent to which clients verbalize arguments against change (resistance) during MI will be inversely related to the degree of subsequent behavior change, and (c) the extent to which clients verbalize change talk (arguments for change) during MI will be directly related to the degree of subsequent behavior change.

MI should be considered among therapeutic approaches based in the “ordinary language” of clients, in that therapeutic discourse is understood on the basis of common interpretations of what clients say about thoughts, feelings, and behavior (Moyers et al., 2007). The empirical relationship between client language and outcomes thus assumes a central role in the suggested impact of MI; yet evidence for this hypothesized relationship is limited. With respect to verbalizations contrary to change, Miller, Benefield, and Tonigan (1993) reported that independent ratings of client interrupting, arguing, off-task responses, and other negative responses within a brief intervention about alcohol use were strongly predictive of poorer 1-year drinking outcomes. Recently, Moyers et al. (2007) coded client language of participants randomly selected from each of three Project MATCH therapy conditions. Rates of both positive and counter–change language in the first clinical session across treatments predicted follow-up drinking outcomes.

Further evidence of a link between positive client change language and clinical outcomes in MI sessions was reported by Amrhein, Miller, Yahne, Palmer, and Fulcher (2003) in sessions with illicit drug users as they presented for treatment. Client language was subjected to a linguistic analysis that subdivided change language into elements thought to vary with respect to the nature and degree of commitment to change (commitment, desire, ability, need, readiness, and reasons). Clients who reduced their substance use during and after treatment made more frequent commitment to change statements during the evaluation of a change plan at the end of the interview compared with those who intermittently used substances over time or who never improved. Other aspects of change language were predictive of commitment language.

This small, emerging evidence base provides some support for the notion that client change language can prospectively predict clinical outcomes. These studies, however, have been completed with adults who were seeking treatment (or a free checkup; Miller et al., 1993). Yet MI has been adopted in numerous clinical and nonclinical contexts (Miller & Rollnick, 2002). One common focus for brief interventions in opportunistic contexts is with young people, who seldom seek services on the basis of their own concerns for health and safety (Baer & Peterson, 2002; Monti, Colby, & O'Leary, 2001). Despite this interest, we are unaware of studies that describe the target of MI sessions, describe youth verbalizations about behavior change, or examine whether these relate to current or future behavior. Such information would seem important given that aspects of adult language are still developing in adolescents (Nippold, 2000; O'Kearney & Dadds, 2005). Hence, adolescents receiving feedback about drug use risks may talk about change very differently than adults seeking treatment.

The current analyses sought to extend prior studies of client language within MI sessions by examining the language of homeless youths who use alcohol and illicit substances and were recruited for a risk reduction program (Baer, Garrett, Beadnell, Wells, & Peterson, 2007) that utilized a brief motivational intervention (BMI). We tested whether global ratings of client behavior in MI sessions, and behavioral ratings of positive and counter–change language and commitment talk, would predict changes in substance use from baseline to 1- and 3-month follow-ups.

Design and Procedures

All procedures were approved by the University of Washington Institutional Review Board. A sample of youths (ages 13–19 years) was recruited from a nonprofit, faith-based drop-in center (Baer et al., 2007) by study counselors. Inclusion criteria included lack of stable housing and report of substance use in the prior 30 days. The 127 youths were randomly assigned to receive four sessions of a BMI (n = 75) or drop-in center treatment-as-usual (n = 52), which included no specific clinical intervention, although case management and other services were available. Only youths receiving BMI are included in current analyses. As MI was not found to be superior to treatment-as-usual in this study (see Baer et al., 2007), and treatment-as-usual received no specific treatment sessions, mediating tests of the function of client change for MI were not possible. Follow-up assessments were conducted via appointments and intercept at the drop-in center and on the streets both 1 month and 3 months postbaseline. Following baseline interviews, youths assigned to BMI stayed for the first of four BMI sessions, with later BMI sessions scheduled in the following 4 weeks. Baseline interviews and the BMI sessions were conducted by a master's-level clinician, with follow-up interviews conducted by alternative project staff. As described elsewhere (Baer et al., 2007), participants received cash for completing study assessments and vouchers redeemable at local retail stores for completed BMI sessions. With consent of participants, baseline interviews and BMI sessions were audiotaped.

Sample description and retention

In the original study attrition was not associated with experimental condition, baseline demographic, or substance use rates. Participants identifying themselves as minority racial group members were retained at slightly higher rates compared with those self-identified as nonminority. Of 75 youths assigned to BMI, 21 were removed from analyses due to extensive missing data or concerns about validity of reported substance use. These included youths who (a) were incarcerated in the 30 days prior to baseline assessment (n = 3) or a follow-up assessment (n = 4), (b) received strongly negative interviewer ratings regarding consistency of their responses (n = 2), (c) refused to be audiotaped (n = 6) or were eliminated because of taping malfunction for at least half of the session (n = 4), (d) attended only highly abbreviated sessions (n = 1), and (e) missed both follow-up assessments (n = 1). No differences were seen between those included versus those excluded in relation to gender, age, or racial or ethnic minority (all ns). At baseline the 21 excluded by these criteria reported less substance use in the previous 30 days than the 54 included (days abstinent M = 14.0, SD = 8.8 vs. M = 7.5, SD = 8.9), t(73) = 2.84, p < .01.

The resulting youth sample for current analysis showed a fairly even gender distribution (54% male vs. 46% female) and mean age of 17.9 years (SD = 1.3). Reported ethnicity was 59% Caucasian, 17% multiracial, 7% Native American, 7% African American, 6% Hispanic or Latino, and 4% Asian or Pacific Islander. The sample reported high rates of substance use, with a mean of 7.5 (SD = 8.9) days of abstinence from substances in the month prior to baseline assessment. Alcohol use in the past month was reported by 89.1% of the sample. Marijuana was the most commonly used drug in the past 30 days by youth report (94.4%), followed by “club drugs” (57.1%), methamphetamine (53.5%), hallucinogens (35.9%), cocaine (33.3%), and opiates other than heroin (33.3%).

Intervention description

The BMI followed the model of an extended substance use “checkup” in which information or exercises about patterns and risks of substance use are provided as personal feedback (Miller, Sovereign, & Krege, 1988). Intervention content is described elsewhere (Baer et al., 2007). Interventionists were trained in MI techniques, then supervised via regular-session audiotape review by John S. Baer (a licensed psychologist, member of the Motivational Interviewing Network of Trainers, and an experienced MI trainer). Interventionists were trained to be nonconfrontational in tone and to provide advice about risk reduction only with permission. Initial sessions averaged 17 min (SD = 7), with subsequent sessions averaging 33 min (SD = 11). Of the 54 youths assigned to BMI, 27 attended all four sessions, 9 attended three, 9 attended two, and 9 attended one. Mean intervention time across sessions was 79.8 min (SD = 42.9). Youth ratings of the intervention were overwhelmingly positive (Baer et al., 2007).

Measures

Substance use frequency and severity

Participants were asked to recall substance use in the prior 30 days through a modified Time Line Follow-Back interview (Sobell & Sobell, 1993). Given that individuals used many combinations of substances over the 30-day period, an omnibus measure of “days of abstinence” from alcohol, marijuana, and other drug use (excluding tobacco) was calculated. We used abstinence days for conceptual clarity because days of use may represent days of use of any number of substances, whereas abstinence is clearly defined by no substance use.

Validation of drug use self-report

Urinalysis was used to calculate sensitivity and specificity of self-reported cocaine, amphetamine, opiate, and marijuana use at the 3-month follow-up. With 41 of 54 youths (76%) providing urinalysis, no evidence of systematic underreporting was observed (Baer et al., 2007).

Indices of intervention discourse

Audiotaped BMI sessions were coded by three trained raters. The scoring system utilized components of the Motivational Interviewing Skill Code (Version 1.0; Miller, 2000; Moyers, Martin, Catley, Harris, & Ahluwalia, 2003) and psycholinguistic framework of Amrhein et al. (2003) to characterize adolescent language in BMI sessions. Based on the Motivational Interviewing Skill Code, the scoring system included three global ratings of client qualities on 7-point Likert scales (1 = not at all, 7 = strong): (a) affect, how openly and directly emotions were expressed; (b) cooperation, how much responsibility for completing in-session tasks was shared; and (c) disclosure, how much personal information was revealed. An additional global score, task orientation, was added to reflect how much the adolescent remained focused and engaged during therapeutic tasks.

In line with Amrhein et al. (2003), frequency of youth statements about change, termed change talk (Miller & Rollnick, 2002), was tallied relative to substance-related risk reduction, promotion of health and safety, or service utilization. Three categories were coded to reflect differing types of language and degree of interest in change noted in prior research (Amrhein et al., 2003) and based on qualities of adolescent language observed in this and previous studies. These included (a) commitment, or statements of explicit intention to change (or maintain) behavior (e.g., “I'm not going to use meth again” or “I'm never going to stop smoking pot”); (b) reasons, or statements providing a rationale for (or against) behavior change (e.g., “When I'm high I fight with my girlfriend” or “Using drugs helps me cope with being on the street”); and (c) desire/ability, or statements that indicate desire, willingness, or ability or self-efficacy to change or not to change (e.g., “I'd like to cut back my drinking” or “I can't deal with not using because I know it'll be too painful”). Irrespective of categorization, each instance of change talk was also assigned a positive or negative valence reflecting if the statement was in favor of or opposed to reduced substance use, reduced health risks, or increased service utilization. Coders recorded tallies of each type of change talk across 5-min intervals within each session. To control for varying degrees of talkativeness, session lengths, and attendance, we converted tallies to average rates of change talk per 5 min. Overall rates of each type of change talk were calculated across individual sessions, or across all sessions attended, as the total tally across 5-min intervals divided by number of 5-min intervals.

Training and Reliability of Raters

Three raters received extensive training in recognizing and rating MI elements in therapeutic dialogue and attended weekly supervision meetings where scoring dilemmas were discussed. Interrater reliability was assessed from a subset of sessions that all three rated (n = 15, 8.3%). Intraclass correlations, computed as a two-way random model, were acceptable on the basis of Cicchetti (1994) criteria (above .40), although lowest for commitment language (see Table 1).

Coding Reliability, Means, and Standard Deviation for Youth and Counselor Language, Averaged Across Sessions

Descriptives for session codes are shown in Table 1. For the 7-point scale, mean global scores for cooperation, affect, and task orientation were between 4 and 5, whereas youth disclosure was just above 5. Youths more often expressed reasons for, reasons against, and desire/ability for change, and less often expressed desire/ability against, commitment for, and commitment against change.

Intercorrelations between global scores, change language, and substance use rates at baseline and follow-up assessments are presented in Table 2. As can be seen, desire/ability comments against change were significantly correlated with substance use rates at follow-up assessments. There also are several modest correlations among global ratings and among rates of use of different forms of change language. Among desire/ability and commitment, respectively, statements for and against change were correlated. This may reflect greater talkativeness or suggest that youths may tend to use one form of change language to express both positive and negative motivation. Partial correlations (controlling for the sum of other talk types) suggest that the former may be true for desire/ability for and against change (r = .35, p < .05 vs. partial r = .23, ns) and the latter true for commitment for and against change (r = .40, p < .01 vs. partial r = .39, p < .01).

Intercorrelations Among Language Codes, Global Ratings, and Substance Use Rates (n = 50–54)

Multiple regression assessed whether session language predicted changes in substance use rates. For each follow-up point, multiple regression predicting substance use abstinence in the prior 30 days was performed in Mplus 5.0 through robust maximum likelihood estimation, for sets of predictors. Three participants were missing assessment at 1 month and 4 at 3 months (all 54 had data for at least one follow-up). To assess prediction of change in substance use rates and control for treatment exposure, we used regressions that included baseline prior 30-day abstinence and the number of feedback sessions attended. Demographic characteristics (gender, race, age) were unrelated to either predictors or outcomes in preliminary analyses and were excluded in regressions to preserve statistical power in this small sample. Because counselors were not associated with substance use outcomes in the original study, they also were excluded in regression analyses. As noted in our prior study (Baer et al., 2007), days of abstinence increased over time for those receiving BMI (not differing from those in the control group). For youths included in current analyses, average days of abstinence over the prior 30 days was 7.5 (SD = 8.9) at baseline, then increased to 11.5 (SD = 10.8) at 1-month follow-up and 11.0 (SD = 10.6) at 3-month follow-up.

Tests of the four youth global scores on change in days of abstinence showed that higher task orientation predicted more days of abstinence at the 3-month but not the 1-month follow-up (standardized β = .44, p < .01). No other effects were statistically significant.

In regressions predicting 1-month abstinence rates from youth verbalizations about change, two of the six youth talk rates had statistically significant effects. Reasons in favor of change predicted greater abstinence, and desire/ability against change predicted less abstinence (standardized β = .23 and −.57, respectively, p < .05 and .001). Only desire/ability against change predicted less abstinence at the 3-month time point (standardized β = −.41, p < .05). Figure 1 illustrates the magnitude of effect based on desire/ability against change. Those who expressed more than the median desire/ability against change reported mean abstinence days at the 1- and 3-month follow-ups of 6.9 and 8.7 (SD = 9.1 and 10.0, respectively), whereas those making less than the median amount reported mean abstinence days of 16.2 and 13.4 (SD = 10.5 and 11.0, respectively).

Figure 1. Days of abstinence depending on desire/ability against change talk (individuals above vs. below the median).

Supplemental analyses addressed possible alternative interpretations of results from regression analyses. Outlier analyses (Cohen, Cohen, West, & Aiken, 2003) in which Cook's distance and leverage scores were computed and outliers were removed showed no substantive changes in these findings. Regressions for 1- and 3-month outcomes including significant predictors from the two original regression analyses (global ratings and rates of change language) suggested that the effect of task orientation on 3-month abstinence did not persist once change language was included. Additional analyses revealed no predictive effects of youth talk in the first BMI session, of changes in rates of language across sessions, or differential effects of change talk rated in the first half versus second half of sessions.

Finally, we explored why reasons for change had a nonsignificant zero-order correlation with 1-month abstinence (r = −.03; see Table 2) but was a significant predictor in multivariate regressions (including those controlling for outlier effects). Review of partial correlations with other variables in the regression model suggested statistical suppression only in relation to desire/ability against change (partial r = .27, p = .065). A 2 × 2 analysis of variance of the relation between these two variables in prediction of change in substance use rates, on the basis of median split of each, suggested an interaction, F(1, 3) = 3.10, p = .09. When desire/ability against change was high, there was a detrimental effect on change in abstinence regardless of whether reasons for change was low or high (mean abstinence change = 1.0 and 0.5, for those with low and high reasons for change, respectively, SD = 5.5 and 7.9); however, when desire/ability against change was low, reasons for change appears to affect abstinence (mean abstinence change = 2.9 and 11.7, for those with low and high reasons for change, respectively, SD = 11.2 and 10.3).

Our examination of youth verbalizations during BMI sessions provides support for one of the basic tenets of MI: that client change language is related to subsequent behavior change. Despite myriad psychological and social problems among homeless youths and their general disengagement from broader social systems, and despite that this sample was not seeking treatment, change language in BMI sessions regarding substance use or service utilization significantly and prospectively predicted changes in substance use. Analyses suggest that two aspects of youth language about change are differentially indicative of actual behavioral change: statements about reasons in favor of change and statements counter to the desire or ability to change.

The strongest effect was noted for desire/ability language against change. Such comments were strongly and prospectively associated with substance use reductions at both follow-ups, despite being fairly uncommon (mean frequency of .61 per 5 min). Thus, data suggest that within BMI sessions with adolescents, a few comments about not wanting, needing, or being able to change bode poorly for subsequent reduction in substance use. Such utterances may be a marker of a specific type of resistance in MI sessions; resistance has been associated with poor clinical outcomes among adolescents (Karver, Handelsman, Fields, & Bickman, 2006; Shirk & Karver, 2003). This finding is also consistent with literature on behavioral intentions (Fishbein & Ajzen, 1975; Fishbein, Hennessy, Yzer, & Douglas, 2003; Webb, Baer, Getz, & McKelvey, 1996), despite these observational codes differing from typical measures of attitudes and behavioral intention.

Stated reasons for change also reveal a significant prospective relation with outcomes, albeit smaller than that noted for comments about desire/ability against change and possibly relevant only when statements of desire/ability against change are absent. Prior research (Baer, Peterson, & Wells, 2004) suggests that most homeless youths describe themselves as precontemplative or contemplative about change. We expected that, at best, homeless youths would express ambivalence about change in substance use. Given the brief intervention process where feedback is provided about risks and counselors strategically work to elicit positive reasons for change, we expected that problem recognition expressed through reasons for change might emerge. Compared with other forms of change language, positive reasons for change were the most common form of verbalization (1.32 per 5-min interval). Still, this rate does not suggest that these BMI sessions were filled with discussions about negative aspects of substance use and benefits of reduction and cessation. A relatively few stated reasons for change may be important for future behavior nonetheless, especially if youths are not otherwise negative with respect to their desire or ability to change. Our data do not suggest that the rate of such language changed across or within sessions.

Commitment language, which Amrhein et al. (2003) have argued as most important in the prediction of substance use outcomes, was not found to be associated with change in this study. Amrhein et al.'s study differed from this one in that it took place among adults who were seeking treatment for drug problems, and the brief MI intervention included a discussion of a change plan at the end. In contrast, our protocol with homeless youths varied from one to four sessions and followed a booklet from which adolescents could choose topics for discussion. In this context, commitment language was rare (0.27 per 5 min in favor of change, 0.18 per 5 min against change), perhaps accurately reflecting the fact that youths were not seeking assistance. Yet an intervention that includes exercises that elicit commitment language (such as developing a change plan) might produce positive findings about commitment language. Additionally, commitment language among these youths proved difficult to code; intraclass correlations were lowest for this category. A different rating method for this language category might lead to different results.

It is important to acknowledge that this study tests the relationship between youth verbalizations and subsequent behavior only and does not test a complete causal chain for MI efficacy. Several additional features of the study should cause readers to draw conclusions cautiously. We would be more confident in these relationships if we had a larger sample and more variability in youth substance use. Despite a careful approach to data analysis (including evaluating the influence of few cases or outliers), the inherent overfitting of regression models requires that the relationships we have reported be replicated. The study is also limited by the use of self-reported substance use (although urine testing suggests that reports were not systematically biased). Results are based on those youths who were not recently incarcerated and who allowed audiotaping. Generalization to other adolescent populations, both housed and homeless, requires study. Results also relied on a specific coding scheme that was informed by prior studies yet altered by investigators for this study. Alternative coding schemes, which either combine or split constructs differently, may suggest different patterns and predictive relationships. Although we analyzed only those codes that reached a minimum level of confidence based on Cicchetti's (1994) standards, higher and more consistent interrater reliability would enhance confidence in the observed relationships.

Although not providing a direct mediating test, the relationships reported here should provide support for those who design brief interventions in which counselors are taught to pay careful attention to and elicit change language from adolescent clients. Youth verbalizations about change may provide one method of assessment for likelihood of change, which could lead to tailored interventions. For the clinician, taping, transcribing, and reliably coding verbalizations are potentially unfeasible. If these results are replicated, attention to more parsimonious means of assessing change talk will allow for clinical application.

Our data suggest that negative comments about desire or ability against change, even when observed infrequently, are relatively strongly predictive of poor outcomes. Additionally, even among youths with multiple social and psychological difficulties, positive statements about reasons for change may indicate that risk reduction is likely. Whether and how the process of language development itself moderates this relationship would be a fruitful area for exploration. In addition, it is left for future research to establish that the specific eliciting and reinforcing of such change language (e.g., through MI) can in fact alter the course of substance use and other risk behavior among high-risk youths.

¹ To assess beliefs about risks and harms associated with substance use, we used codes for reasons for and reasons against change that included statements referring to other people in addition to self. Analyses were completed with and without such codes without appreciable differences in results.

² Despite lower coding reliability, scores for commitment language were included in initial predictive analyses. Study results did not differ with subsequent exclusion of commitment language from analyses.

Amrhein, P. C., Miller, W. R., Yahne, C. E., Palmer, M., & Fulcher, L. (2003). Client commitment language during motivational interviewing predicts drug use outcomes. Journal of Consulting and Clinical Psychology, 71(5), 862–878.

Baer, J. S., Garrett, S. B., Beadnell, B., Wells, E. A., & Peterson, P. (2007). Brief motivational intervention with homeless adolescents: Evaluating effects on substance use and service utilization. Psychology of Addictive Behaviors, 21(4), 582–586.

Baer, J. S., & Peterson, P. L. (2002). Motivational interviewing with adolescents and young adults. In W. R.Miller & S.Rollnick (Eds.), Applications of motivational interviewing (2nd ed.). New York: Guilford Press.

Baer, J. S., Peterson, P. L., & Wells, E. A. (2004). Rationale and design of a brief substance use intervention for homeless adolescents. Addiction Research & Theory, 12(4), 317–334.

Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290.

Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analyses for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.

Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention, and behavior. Reading, MA: Addison-Wesley.

Fishbein, M., Hennessy, M., Yzer, M., & Douglas, J. (2003). Can we explain why some people do and some people do not act on their intentions?Psychology, Health & Medicine, 8(1), 3–18.

Hettema, J., Steele, J., & Miller, W. R. (2005). Motivational interviewing. Annual Review of Clinical Psychology, 1(1), 91–111.

Karver, M. S., Handelsman, J. B., Fields, S., & Bickman, L. (2006). Meta-analysis of therapeutic relationship variables in youth and family therapies: The evidence for different relationship variables in the child and adolescent outcome literature. Clinical Psychology Review, 26(1), 50–65.

Miller, W. R. (2000). Motivational Interviewing Skill Code (MISC): Coder's manual. Retrieved from http://casaa.unm.edu/download/misc1.pdf

Miller, W. R., Benefield, R. G., & Tonigan, J. S. (1993). Enhancing motivation for change in problem drinking: A controlled comparison of two therapist styles. Journal of Consulting and Clinical Psychology, 61(3), 455–461.

Miller, W. R., & Rollnick, S. (2002). Motivational interviewing: Preparing people to change addictive behavior (2nd ed.). New York: Guilford Press.

Miller, W. R., Sovereign, R. G., & Krege, B. (1988). Motivational interviewing with problem drinkers: II. The Drinker's Check-up as a preventive intervention. Behavioural Psychotherapy, 16(4), 251–268.

Monti, P. M., Colby, S. M., & O'Leary, T. A. (2001). Adolescents, alcohol, and substance abuse: Reaching teens through brief interventions. New York: Guilford Press.

Moyers, T. B., Martin, T., Catley, D., Harris, K. J., & Ahluwalia, J. S. (2003). Assessing the integrity of motivational interviewing interventions: Reliability of the motivational interviewing skills code. Behavioural and Cognitive Psychotherapy, 31(2), 177–184.

Moyers, T. B., Martin, T., Christopher, P. J., Houck, J. M., Tonigan, J. S., & Amrhein, P. C. (2007). Client language as a mediator of motivational interviewing efficacy: Where is the evidence?Alcoholism: Clinical and Experimental Research, 31(Suppl. 3), 40S–47S.

Nippold, M. A. (2000). Language development during the adolescent years: Aspects of pragmatics, syntax, and semantics. Topics in Language Disorders, 20(2), 15–28.

O'Kearney, R., & Dadds, M. R. (2005). Language for emotions in adolescents with externalizing and internalizing disorders. Development and Psychopathology, 17(2), 529–548.

Rollnick, S., & Miller, W. R. (1995). What is motivational interviewing?Behavioural and Cognitive Psychotherapy, 23(4), 325–334.

Shirk, S. R., & Karver, M. S. (2003). Prediction of treatment outcome from relationship variables in child and adolescent therapy: A meta-analytic review. Journal of Consulting and Clinical Psychology, 71(3), 452–464.

Sobell, M. B., & Sobell, L. C. (1993). Treatment for problem drinkers: A public health priority. In J. S.Baer & G. A.Marlatt (Eds.), Addictive behaviors across the life span: Prevention, treatment, and policy issues (pp. 138–157). Newbury Park, CA: Sage Publications.

Webb, J. A., Baer, P. E., Getz, J. G., & McKelvey, R. S. (1996). Do fifth graders' attitudes and intentions toward alcohol use predict seventh-grade use?Journal of the American Academy of Child & Adolescent Psychiatry, 35(12), 1611–1617.

Submitted: October 5, 2007 Revised: April 4, 2008 Accepted: April 23, 2008

Adolescents With Suicidal and Nonsuicidal Self-Harm: Clinical Characteristics and Response to Therapeutic Assessment

Acknowledgement: This research was supported by grants from Psychiatry Research Trust and West London Primary Care Research Consortium. We thank Gordana Milavic, Jo Fletcher, Paul Calaminus, Azeem Majeed, Robert Goodman, and Derek Bolton for their clinical, managerial, and research support.

Self-harm in adolescence is a common problem (Evans, Hawton, Rodham, & Deeks, 2005) with lifetime prevalence of attempted suicide of 9.7%, whereas an additional 13.2% of adolescents engage in self-harm at some point during that period. Self-harm is one of the strongest predictors of eventual death by suicide in adolescence, increasing the risk up to 10-fold (Hawton & Harriss, 2007).

Despite its high prevalence, there is no agreement about the definition of self-harm. Hawton et al. (2003) defined (deliberate) self-harm as intentional self-injury or self-poisoning, irrespective of type of motivation or degree of suicidal intent. Many European investigators use this definition (Hawton et al., 2003; Schmidtke et al., 1996), which is also used in Australia (De Leo & Heller, 2004) and New Zealand (Carter, Reith, Whyte, & McPherson, 2005). Many American researchers subdivide self-harm into two main groups, suicidal acts with intent to die and instrumental suicide-related behavior with no intent to die (Silverman, Berman, Sanddal, O'Carroll, & Joiner, 2007), and have proposed revisions to the Diagnostic and Statistical Manual of Mental Disorders disorders and criteria to include the new category Non-Suicidal Self Injury (American Psychiatric Association, 2010). Although there is much debate on this issue, few studies have investigated differences between patients with suicidal and nonsuicidal self-harm.

If differences between the two groups exist, they could be related to one of these three domains: demographic characteristics, clinical characteristics, and self-harm-related variables (Walsh, 2006). Most of the proposed differences have not been demonstrated in large population-based studies directly comparing suicidal versus nonsuicidal groups.

Very few studies directly compare sociodemographic characteristics of suicide attempters versus nonsuicidal self-harmers (Muehlenkamp, 2005; Nock, Joiner, Gordon, Lloyd-Richardson, & Prinstein, 2006; Sarkar, Sattar, Gode, & Basannar, 2006; Walsh, 2006). In the largest study investigating these differences, those with suicidal self-harm were more likely to be young women with less education from the Southern and Western United States (Nock & Kessler, 2006). A significant fact with respect to the design of this study was that the sample was drawn from young people who all initially classified their behavior as suicidal. However, nearly half of those were subsequently reclassified as having carried out a “suicidal gesture,” that is, self-harm without true suicidal intent.

There are suggestions that nonsuicidal self-harm is equally prevalent among young men and young women (Gratz, 2001; Muehlenkamp & Gutierrez, 2007), although other studies found a female preponderance (Ross & Heath, 2002). Other possible sociodemographic differences may include a higher prevalence of nonsuicidal self-harm in Caucasians as compared with other racial groups (Muehlenkamp & Gutierrez, 2004, 2007).

Several studies propose that those who attempt suicide tend to favor high-lethality methods such as self-poisoning, whereas those who self-harm without suicidal intent are more likely to use low-lethality methods such as self-cutting (Csorba, Dinya, Plener, Nagy, & Páli, 2009; Favazza & Conterio, 1989; Walsh & Rosen, 1988). However, no method of self-harm is exclusively related to suicidal intent, and adolescents may use different methods of self-harm at different times (Nock et al., 2006). Up to 70% of adolescents who self-harm without suicidal intent also attempt suicide (Nock et al., 2006). Guertin, Lloyd-Richardson, Spirito, Donaldson, and Boergers (2001) viewed nonsuicidal self-harm as a complicating factor in suicidal self-harm.

Regarding chronicity, some studies indicate that adolescents who self-harm with no suicidal intent, compared with adolescents with suicidal intent, are more likely to engage in repetitive self-harming behavior (Csorba et al., 2009; Pattison & Kahan, 1983). However, an important minority of adolescents who repeatedly attempt suicide has also been described (Hawton & Harriss, 2008). Up to 55% of adolescents with nonsuicidal self-injury also repeatedly attempt suicide (Nock et al., 2006).

There are no consistent findings differentiating suicidal and nonsuicidal self-harm in terms of time of onset. The frequency of both behaviors increases in adolescence and young adulthood and then diminishes over time. However, the peak onset for nonsuicidal self-harm could be between 12 and 14 years (Muehlenkamp & Gutierrez, 2004; Ross & Heath, 2002), whereas for suicidal self-harm, the peak onset is around the age of 16 (Nock et al., 2008).

Adolescents with both suicidal and nonsuicidal self-harm overwhelmingly (in about 90% of cases) satisfy diagnostic criteria for one or more psychiatric disorders (Jacobs, 1999; Nock et al., 2006), depression being the most common diagnosis in both groups. These results, however, only apply to the clinical samples, and it may be that in the nonreferred adolescents, the rate of psychiatric disorders is much lower. Nock and Kessler (2006) also found that adolescents with suicidal versus nonsuicidal self-harm have a higher prevalence of depression, drug abuse and drug dependence, conduct disorder and antisocial personality disorder, phobias, and multiple diagnoses. Limitations of the methodology of this study were discussed above; however, the results are important in the light of other studies that found differences between the two groups in depression scores, suicidal thinking, and attitude to life (Csorba et al., 2009; Jacobson, Muehlenkamp, Miller, & Turner, 2008; Muehlenkamp & Gutierrez, 2004), as well as the prevalence of anxiety disorders (Brausch & Gutierrez, 2010; Csorba et al., 2009; Jacobson et al., 2008; Muehlenkamp & Gutierrez, 2004). Regarding conduct problems, there have also been reports of a higher prevalence of externalizing disorders in nonsuicidal adolescents with self-harm (Grøholg, Ekeberg, & Haldorsen, 2000).

At present, most random allocation studies do not distinguish between adolescents with suicidal and nonsuicidal self-harm (Chanen et al., 2008; Cotgrove, Zirinsky, Black, & Weston, 1995; Harrington et al., 1998; Hazell et al., 2009; Wood, Trainor, Rothwell, Moore, & Harrington, 2001). In addition, despite psychosocial assessment having distinct therapeutic value (Poston & Hanson, 2010), no studies demonstrate a differential response of adolescents with different self-harm subtypes to a brief therapeutic intervention. There is good evidence of therapeutic response to a combination of medication and psychotherapy in young people with depression, including those with self-harm (Brent et al., 2008, 2009; TADS Team, 2007).

The literature on subtypes of adolescents with self-harm is limited and developing. To investigate the possible differences between the adolescents with suicidal and nonsuicidal self-harm, we studied the participants of the Trial of Therapeutic Assessment in London (TOTAL; Ougrin et al., 2011). In the TOTAL, 26 clinicians were randomized to deliver either therapeutic assessment (TA), a 30-min manualized intervention in addition to assessment as usual (AAU), or AAU alone. There were 73 adolescents assessed; 70 (96%) of those agreed to participate in the study. Twenty (57%) adolescents in the TA and 18 (51%) in the AAU arm were assessed jointly with their parent or guardian. An average of five adolescents were assessed by each clinician in the TA arm (range = 1–12) and an average of 4.4 (range = 1–14) were assessed in the AAU group. The mean age of the participants was 15.6 years (SD = 1.4). The majority of the participants in the sample were White (n = 37, 53%). Fifty-six (80%) participants were young women, 28 (40%) self-harmed by self-poisoning alone, 37 (53%) self-harmed by self-injury alone, and 5 (7%) self-harmed by both self-poisoning and self-injury. Forty-one (59%) had self-harmed previously. There was no statistically significant difference between the TA and AAU arms in the proportion of the adolescents presenting with suicidal self-harm (31% vs. 34%, χ² = .065, p < .8). Overall, 53 (76%) participants met diagnostic criteria for at least one diagnosis, 42 (60%) had a primary diagnosis of an emotional disorder, nine (13%) had a primary diagnosis of a disruptive disorder, and two (3%) had other diagnoses. The adolescents in the TA arm were significantly more likely than those in the AAU arm to attend the first treatment appointment following the assessment, 29 (83%) versus 17 (49%), odds ratio (OR) = 5.12, 95% confidence interval (CI) [1.49, 17.55], and more likely to attend four or more treatment sessions, 14 (40%) versus 4 (11%), OR = 5.19, 95% CI [2.22, 12.10]. Three months after the initial assessment, there were no statistically significant differences between the intervention and the control arms on the Strengths and Difficulties Questionnaire (SDQ) scores, 15.6 versus 16.0, mean difference = −0.37, 95% CI [−3.28, 2.53], or the Children's Global Assessment Scale (CGAS) scores, 64.6 versus 60.1, mean difference = 4.49, 95% CI [−0.98, 9.96]. Only 34 (49%) of the parent-rated SDQs were available at follow-up; hence, only patient-rated versions were analyzed (available for 63 [90%] of the adolescents). The intervention and the control arms did not differ on repetition of self-harm: 9/35 (26%) in the TA group and 9/35 (26%) in the AAU group, χ² = .0, p = 1.

Using the TOTAL sample, on the basis of the available literature, we hypothesized that adolescents with suicidal self-harm, compared with those in the nonsuicidal self-harm group, will be more likely to be young women and depressed. They also will be more likely to use self-poisoning as a method of self-harm and more likely to benefit from TA as measured by the CGAS and the SDQ scores. We also hypothesized that adolescents in the nonsuicidal self-harm group, compared with those in the suicidal self-harm group, will be more likely to have a diagnosis of conduct disorder, will have an earlier age of onset of self-harm, and will be more likely to have a history of previous self-harm.

In summary, there are significant gaps in the knowledge of the possible differences between adolescents with suicidal and nonsuicidal self-harm, yet these differences may be important in the classification, phenomenology, and eventually differential interventions for the two groups. In this article, we investigate the clinical and sociodemographic differences between the adolescents presenting with suicidal and nonsuicidal self-harm.

Participants

Adolescents 12 to 18 years old not currently engaged with psychiatric services who had self-harmed and been referred for a psychosocial assessment met the inclusion criteria and were eligible for participation in the trial. Exclusion criteria were gross reality distortion (e.g., owing to psychotic illness or intoxication), known history of moderate or severe learning disability, lack of fluent English, immediate risk of violence or suicide, and the need for inpatient psychiatric admission. Self-harm was defined as self-injury or self-poisoning irrespective of the underlying intent (National Institute for Health and Clinical Excellence [NICE], 2004), in line with British national guidelines.

The referral for psychosocial assessment was made either following a screening at the emergency departments of four inner London hospitals or following an urgent family doctor's referral to the local outpatient child and adolescent mental health services. Both the referring practitioner and the emergency department staff were blind to the allocation of the adolescents to either TA or AAU.

Eligible participants were approached with respect to participating in the trial after they had received medical clearance after an episode of self-harm. Parents (and 16- or 17-year-old participants) signed an informed consent document and adolescents younger than 16 years of age assented to participate. All participants and their guardians, if present, completed the SDQ and were assigned a CGAS score.

Interventions and Procedure

Control arm: Assessment as usual

AAU included a standard psychosocial history and risk assessment and followed the recommendations set out in the NICE (2004) guidelines. The assessment letter was sent to the relevant community team, and a copy was sent to the family in accordance with the trusts' policies.

A random sample of 10 (29%) of the clinical evaluations was selected to evaluate fidelity. Two independent psychiatrists rated adherence to 17 points that should be covered in a standard clinical self-harm evaluation (NICE, 2004). Adherence to these 17 points averaged 81.2%, with a minimum of 71% and a maximum of 100%. Interrater agreement was acceptable (overall Cohen's κ = .73, p < .001, range: .43–1).

Intervention arm: TA

The major components of TA are as follows.

1. Assessing risk and taking a standard psychosocial history (approximately 1 hr).
2. Taking a 10-min break to review the information gathered and to prepare for the rest of the session, followed by a 30-min intervention covering the next four steps.
3. Jointly constructing a diagram (based on the cognitive analytic therapy paradigm) that consists of three elements: reciprocal roles, core pain, and maladaptive procedures (Ryle, 2010).
4. Identifying a target problem.
5. Considering and enhancing motivation for change.
6. Exploring potential exits (i.e., ways of breaking the vicious cycles identified).
7. Describing the diagram and the exits in an “understanding letter.” In addition to the understanding letter, the family also receives the usual assessment letter.

The assessment process was manualized, although assessing clinicians used clinical judgment when deciding on the best approach to exits. Family members were involved in all stages of TA whenever possible. Clinicians received training in TA over five half days accompanied by weekly homework and a video assessment before and after training with independent fidelity assessment.

To ensure fidelity to TA, we selected a random sample of 10 (29%) of the clinicians' audiotaped evaluations. Two independent clinicians rated adherence to the seven components of TA. Adherence to these seven points averaged 90.7%, with a minimum of 71.4% and a maximum of 100%. Interrater agreement was moderate (overall Cohen's κ = .64, p < .001, range: .59–1). All 10 tapes showed that the clinicians had achieved the required level of competence in TA (33 points or more on the objective subscale of the Therapeutic Assessment Quality Assurance Tool, a 0–50 scale completed by an independent clinician rating the extent to which the five objectives of TA had been achieved).

Three months after the initial assessment, three higher specialist trainees in psychiatry unaware of the patients' allocation, conducted face-to-face interviews with the participants and their guardians, if available. If a face-to-face interview was not possible, a telephone follow-up interview was conducted. Participants and their guardians, whenever available, also completed the follow-up version of the SDQ.

Randomization occurred at the level of clinician, and 26 clinicians were randomized. Twenty-two clinicians were from Center 1, and two each were from Centers 2 and 3. Randomization was conducted by a senior psychiatrist independent of the study clinicians. The randomization was stratified by center, and two blocks (block lengths 22 and 4) were created using a permuted block design to ensure that equal numbers of clinicians from each center were allocated to either intervention or control groups. The randomization scheme was generated using Web-based randomization software (http://www.randomization.com). The clinicians were informed of their allocation by e-mail. Once randomized, the clinicians administered either TA or AAU to all eligible adolescents presenting with self-harm for assessment as part of their routine work until a total of 70 adolescents were recruited. Randomization occurred irrespective of the type of self-harm in the adolescents.

Power calculation for the TOTAL study (Ougrin et al., 2011) was based on the results of a pilot study (Ougrin, Ng, & Low, 2008). It was assumed that 75% of the participants in the intervention (TA) group and 40% of the participants in the control (AAU) group would attend the first community treatment session. The software nQuery Advisor 4.0 (Elashoff, 2000) was used to establish that 35 participants in each group (70 in total) were required. With the sample size of 70 and 80% power, differences between the two groups equal to or greater than an effect size of .48 for continuous variables and equal to or greater than an odds ratio of 2.79 for dichotomous variables at the two-sided 5% level were detectable.

It was not possible to keep the clinicians unaware of the intervention they were delivering. Participants were unaware as to what type of assessment they were receiving. The study statistician and the researchers conducting follow-up assessments were unaware of the participants' allocation.

Measures

Clinical diagnosis

Clinical diagnosis was recorded using the International Classification of Diseases 10th edition (ICD–10) criteria (World Health Organization, 1992). To aid clinicians with the diagnostic process, the electronic Patient Journey System was used in 66 of the 70 assessments. The system provides examples of questions based on ICD–10 criteria for a range of psychiatric disorders and requires that clinicians arrive at a primary diagnosis. Primary clinical diagnoses were collapsed to form the following four groups: (a) no diagnosis, (b) emotional disorders (including depressive and anxiety disorders), (c) disruptive disorders (including conduct and hyperkinetic disorders), and (d) other disorders.

Sociodemographic data

Basic demographic data were collected using the standard self-report forms used for the assessment of all new patients at the participating National Health Service hospital trusts. Ethnicity was recorded on the basis of a self-report question. The patients had to initially choose from 16 different categories. These were subsequently collapsed into two: White or non-White. The parental marital status question initially had eight response categories. These were collapsed into either intact (living with two biological parents) on nonintact.

A rating of socioeconomic status was assigned by the assessing clinician on the basis of the occupation of the main breadwinner in the family (Office of Population Censuses and Surveys, 1991): Social Class I = higher professional and managerial occupations; Social Class II = intermediate managerial, administrative, or professional; Social Class III A = supervisory or clerical and junior managerial, administrative, or professional; Social Class III B = skilled manual workers; Social Class IV = semiskilled and unskilled manual workers; and Social Class V = casual or lowest grade workers, pensioners, and others who depend on the state for their income. This scale was also collapsed into two categories: manual and nonmanual workers.

Suicidality

The Columbia Classification Algorithm of Suicide Assessment (Posner, Oquendo, Gould, Stanley, & Davies, 2007) was used to classify the young people on the basis of the index episode of self-harm. This is a classification algorithm in which the criteria for a suicide attempt include both self-injurious behavior and suicidal intent (at least some intention to commit suicide). The Columbia Classification Algorithm of Suicide Assessment has been shown to have good validity and reliability (interclass correlation coefficient = .89). Because an episode of self-harm was an inclusion criterion for TOTAL, we used the following three categories to classify each index episode: (a) suicide attempt; (b) self-injurious behavior, no suicidal intent; and (c) self-injurious behavior, suicidal intent unknown. The clinical description of each index episode was independently rated by two senior psychiatrists. Disagreements were resolved by consensus. The three categories were then dichotomized into suicidal self-harm, which included Categories a and c, and nonsuicidal self-harm. Clinical guidelines suggest that the assessing clinicians should err on the side of caution when assessing adolescents with self-harm. The presence of suicidal intent in the adolescents with unknown intent should therefore be assumed (American Academy of Child & Adolescent Psychology, 2001; NICE, 2004).

Measures of general psychopathology and function

Self-rated and parent-rated versions of the SDQ were used. The SDQ (Goodman, 2001) consists of 25 items that make up five 5-item subscales assessing conduct problems, hyperactivity–inattention, emotional symptoms, peer problems, and prosocial behavior. Each item is rated on a 3-point scale: not true (a score of 0), somewhat true (a score of 1), or certainly true (a score of 2). The total difficulty score ranges from 0 to 40, and the Prosocial Behavior subscale score is not included in the total. A score of 20 or more indicates clinical abnormality. Only adolescent-reported SDQs were used, as only 39 (56%) of the adolescents had parental baseline scores.

The CGAS was also used. The CGAS (Shaffer et al., 1983) is a clinician-rated scale assessing the functional status of a young person on a scale of 1–100, with a score of at least 70 being indicative of adequate functioning.

Statistical Analysis

Comparisons between the adolescents randomized to TA versus AAU are described in Ougrin et al., 2011. Here, all comparisons were conducted between the adolescents with suicidal and nonsuicidal self-harm.

To evaluate the impact of TA, we used a random effects model with the intervention (TA vs. AAU), suicidality (suicidal vs. nonsuicidal self-harm), and the interaction between the intervention and suicidality as independent factors and CGAS and SDQ change scores as dependent variables. Patients were not sampled individually but as a group treated by a clinician. To adjust for possible clinicians' effects, we included clinician as a random factor in the model that explicitly models the possible correlation between patients of the same clinician. We chose the CGAS and SDQ change score, from baseline to follow-up, as a dependent variable. We selected this design because the comparison groups used in this study (adolescents with and without suicidal intent) were different from the intervention and control arms created by the original randomization (adolescents allocated to TA vs. AAU irrespective of the intent; Ougrin et al., 2011), and we could not therefore use the analysis of covariance approach to analyzing clustered randomized controlled trials.

There was a good level of agreement between the two raters in classifying suicidal and nonsuicidal cases (κ = .69, p < .001). The disagreements were resolved by consensus.

There were no significant differences between the two groups on most demographic characteristics studied. Suicidal adolescents were more likely to be young women (see Table 1).

Demographic Characteristics of Adolescents With Suicidal and Nonsuicidal Self-Harm

Of the clinical characteristics, the following two variables were found to be significantly different between the two groups. Young people with suicidal self-harm were more likely to use self-poisoning as the method of self-harm, and they were more likely to start self-harming at an older age (see Table 2).

Clinical Characteristics of Adolescents With Suicidal and Nonsuicidal Self-Harm

The following two factors were entered into logistic regression: age of onset and method of self-harm, adjusted for clustering around clinicians, age, and sex. Both method of self-harm and age of onset were significantly associated with type of self-harm. Adolescents with self-poisoning were more likely to belong to the suicidal self-harm group. Adolescents with suicidal self-harm were more likely to have a later age of onset of self-harm (see Table 3).

Summary of Hierarchical Regression Analysis for Variables Predicting Differences Between Adolescents With Suicidal and Nonsuicidal Self-Harm

Impact of TA

In addition, we studied the impact of having TA versus AAU on two main clinically relevant findings: the SDQ and the CGAS scores. The SDQ and the CGAS score change from baseline to the 3-month follow-up assessment were used.

A random effects model with the intervention as an independent factor and the clinician as a random factor showed that there was a significant difference in the intervention effect on the CGAS score change in the nonsuicidal group but not in the suicidal group. For the mean difference between TA and AAU in the suicidal group, M = 0.22, 95% CI [−10.66, 11.11], t(18) = 0.043, p = .966; for the mean difference between TA and AAU in the nonsuicidal group, M = 6.39, 95% CI [0.67, 12.11], t(43) = 2.25, p = .03. However, this difference in change score between the suicidal and nonsuicidal groups was accompanied by a nonsignificant interaction between intervention and suicidality: F(1, 61) = 1.27, p = .26 (two factorial random effects model with intervention [TA vs. AAU] and type of self-harm [suicidal vs. nonsuicidal] as fixed factors).

Suicidal and nonsuicidal adolescents did not differ on the SDQ score change at the 3-month follow up. Random effects models with the intervention as an independent factor and the clinician as a random factor showed no significant difference in the SDQ score change between TA and AAU in either the nonsuicidal group or the suicidal group. For the mean difference between AAU and TA in the suicidal group, M = 3.12, 95% CI [−1.49, 7.73], t(18) = 1.42, p = .172; for the mean difference between AAU and TA in the nonsuicidal group, M = −0.42, 95% CI [−3.56, 2.72], t(41) = 0.27, p = .79.

A two-factorial random effects model did not reveal a significant interaction between type of intervention and suicidality, F(1, 59) = 1.68, p = .20, with the change of the SDQ score as the dependent variable and intervention (TA vs. AAU) and type of self-harm (suicidal vs. nonsuicidal) as independent variables. Furthermore, there was no significant main effect of type of intervention, F(1, 59) = 0.98, p = .33, and suicidality, F(1, 59) = 0.46, p = .50.

Nonsuicidal Self-Injury

We repeated all of the analyses with the nonsuicidal self-injury group (n = 33) compared with other types of self-harm group (n = 30) and separately compared with the suicide attempt group (n = 23). The outcome of these analyses closely resembled the above results and no additional associations were found.

We found three main differences between the adolescents presenting with suicidal and nonsuicidal self-harm in this study: (a) Adolescents with suicidal self-harm tend to have a later age of onset of this behavior, (b) they tend to use self-poisoning more often as the self-harm method, and (c) they are more likely to be young women. In addition, they may respond to TA less favorably than adolescents with nonsuicidal self-harm do. We did not find any differences in other demographic or clinical characteristics. To our knowledge, this is the first study in which the differential response of these two groups to a brief therapeutic intervention at the point of initial assessment was investigated.

Many descriptive studies sought to compare adolescents with nonsuicidal self-harm and those with a combination of suicidal and nonsuicidal self-harm (Brausch & Gutierrez, 2010; Claes et al., 2010). Whereas it is certainly true that significant overlap may exist between these two behaviors, we chose to classify the adolescents in our study on the basis of the index episode of self-harm, in line with other researchers (Grøholg et al., 2000). It is possible that some of the adolescents had a history of both suicidal and nonsuicidal self-harm, yet retrospective assessments of suicidality are unreliable and the index episode approach is the closest to clinical practice.

Differences in Sociodemographic Characteristics

Many authors report self-harm in general to be more common in young women than in young men (Gratz, 2001; Muehlenkamp & Gutierrez, 2007), unlike completed suicide (Ougrin, Banarsee, Dunn-Toroosian, & Majeed, 2010), although this finding is not universal. Several studies also found a female preponderance in those with suicidal versus nonsuicidal self-harm (Muehlenkamp & Gutierrez, 2004; Nock & Kessler, 2006). This study supports the previous findings of a female preponderance in the risk of self-harm overall and in suicidal versus nonsuicidal self-harm in particular. It is possible that community samples with a lower severity of self-harm have a different sex ratio to clinical samples (Madge et al., 2008).

Differences in Self-Harm Characteristics

Our findings are in line with other studies and suggest a strong link between cutting and nonsuicidal self-harm on one hand and self-poisoning and suicidal self-harm on the other (Csorba et al., 2009). However, we did not find an increased likelihood of repeat self-harm in the nonsuicidal group, which was contrary to the findings of other studies (Brausch & Gutierrez, 2010; Csorba et al., 2009).

There is increasing evidence of self-harm being one part of a continuum of suicidal process ranging from thoughts of self-harm to completed suicide (Stanley, Winchel, Molcho, & Simeon, 1992). One longitudinal study demonstrated that nonsuicidal self-harm predicts future nonsuicidal self-harm rather than future suicidal attempts (Wichstrøm, 2009), whereas another demonstrated nonsuicidal self-harm to be a strong predictor of suicide attempts (Wong, 2007). Our findings of nonsuicidal self-harm starting earlier in life than suicidal self-harm are of interest in the light of the theory proposing that suicide is often a result of the learned ability to hurt oneself (Joiner, 2005). Contrary to our expectations, we did not find a greater likelihood of having had a previous episode of self-harm in the nonsuicidal group as compared with the suicidal group.

Differences in Clinical Characteristics

We did not find any differences in the likelihood of having a psychiatric diagnosis between the two groups, although well over 70% of the adolescents satisfied criteria for a psychiatric diagnosis in both groups. As other studies have reported a greater prevalence of psychiatric disorders (up to 90%), it may be that these were underestimated in this study, perhaps because we did not use a full semistructured interview to establish the diagnosis. We did not find an increased likelihood of the diagnosis of depression in the suicidal group, unlike previous studies (Brausch & Gutierrez, 2010; Csorba et al., 2009). However, depression was the most common diagnosis, and the absolute difference between the two groups was large, with 49% of nonsuicidal and 69% of suicidal adolescents satisfying criteria for a depressive condition.

Previous studies have associated more externalizing problems with nonsuicidal self-harm (Grøholg et al., 2000). All adolescents with conduct disorder were in the nonsuicidal group in this study sample. The difference between the two subgroups, however, did not reach a statistically significant level, possibly because of low statistical power. The absolute difference in the diagnosis of conduct disorder was substantial and possibly clinically significant.

Differences in Response to TA

We found no difference in response as measured by the SDQ, but we found that adolescents with nonsuicidal self-harm were more likely to have a higher CGAS score 3 months after the initial TA. However, the differences found were no longer significant when the Intervention × Suicidality interaction was considered, which may be explained by insufficient power. This finding was contrary to our hypothesis. We reasoned that the suicidal group would have a greater risk of having a diagnosis of depression and would therefore have a better response to community treatment on the basis of the combination of psychological treatment and pharmacological intervention (TADS Team, 2007). However, neither of these hypotheses was confirmed. It may be that the absence of clinical improvement in the suicidal group was partly mediated by nonsignificant differences in the prevalence of psychiatric disorders and partly reflected a more refractory nature of psychiatric disorders in this group. Other authors also noted that those with suicidal self-harm are likely to have worse posttreatment outcomes than nonsuicidal comparison groups (Dougherty et al., 2009) and that there is poor treatment engagement in the adolescents with suicidal and nonsuicidal self-harm (Ougrin & Latif, 2011). Both measures of clinical outcome used in this study are rather broad and may have therefore obscured important differences in specific clinical subdomains. Moreover, the 3-months follow-up period may have been too short to allow for the full benefit of the interventions to be realized.

Nonsuicidal Self-Injury

It would appear from the analysis that the nonsuicidal self-injury group closely resembles the nonsuicidal self-harm (which includes adolescents with both nonsuicidal self-poisoning and nonsuicidal self-injury) group. Using the term nonsuicidal self-harm may be preferable to nonsuicidal self-injury, as it encompasses those with nonsuicidal self-poisoning who would have otherwise fallen through the dichotomized classification of nonsuicidal self-injury versus suicide attempts.

Limitations

We did not investigate other important differences that may exist between the adolescents with suicidal and nonsuicidal self-harm. In particular, we did not investigate possible differences in hopelessness, impulsiveness, disordered eating, problem-solving skills, and parental support, all of which have been shown to be potentially different between the two groups (Boergers, Spirito, & Donaldson, 1998; Grøholg et al., 2000; Hargus, Hawton, & Rodham, 2009) and also potentially important for treatment choices. Hopelessness is especially important, as it appears to predict future suicide in those with self-harm, yet the reports from previous studies have not been consistent in establishing the differences between the two subgroups (Brausch & Gutierrez, 2010; Muehlenkamp & Gutierrez, 2007). In addition, our sample size was small, potentially contributing to a low power to detect differences between the two groups.

We did not use a semistructured diagnostic schedule to establish the diagnosis, and this may have contributed to the underestimation of the prevalence of clinical diagnoses in this sample. However computerized prompting integrated with the electronic patients' record system contributed to the standardization of the diagnoses. Although it is an important limitation of the study, using clinical diagnosis has the benefit of generalizing to clinicians' routine practice.

We considered adolescents with clear suicidal intent and those with unclear intent to belong to one group as opposed to combining the latter with the adolescents having clear nonsuicidal intent. This division is clearly arbitrary; however, we reasoned that the young people with mixed or unclear intent should probably be treated as suicidal in clinical practice.

Suicidality rating was done by two independent raters. Neither was blind to the study hypotheses; however, the description of the index episode of self-harm was anonymized and the raters had no access to any other information about the adolescents at the time of the rating. Finally, as mentioned above, the short follow-up period may have prevented the discovery of further differences in treatment response between the two groups.

Clinical Implications

Adolescents with nonsuicidal and suicidal self-harm differ on several important clinical and sociodemographic characteristics and may respond to treatment differently. TA may be especially important in the nonsuicidal group. The nonsuicidal self-injury group seems to be similar to a broadly defined nonsuicidal self-harm group, which includes both nonsuicidal self-poisoning and nonsuicidal self-injury. Clinicians should consider suicidal intent carefully during their initial assessment.

American Academy of Child & Adolescent Psychology. (2001). Summary of the practice parameters for the assessment and treatment of children and adolescents with suicidal behavior. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 495–499. doi:10.1097/00004583-200104000-00024

American Psychiatric Association. (2010). DSM–5: The future of psychiatric diagnosis. Retrieved August 12, 2010, from http://www.dsm5.org/Pages/Default.aspx

Boergers, J., Spirito, A., & Donaldson, D. (1998). Reasons for adolescent suicide attempts: Associations with psychological functioning. Journal of the American Academy of Child & Adolescent Psychiatry, 37, 1287–1293. doi:10.1097/00004583-199812000-00012

Brausch, A. M., & Gutierrez, P. M. (2010). Differences in non-suicidal self-injury and suicide attempts in adolescents. Journal of Youth and Adolescence, 39, 233–242. doi:10.1007/s10964-009-9482-0

Brent, D., Emslie, G., Clarke, G., Wagner, K. D., Asarnow, J. R., Keller, M., . . .Zelazny, J. (2008). Switching to another SSRI or to venlafaxine with or without cognitive behavioral therapy for adolescents with SSRI-resistant depression: The TORDIA randomized controlled trial. JAMA: The Journal of the American Medical Association, 299, 901–913. doi:10.1001/jama.299.8.901

Brent, D. A., Greenhill, L. L., Compton, S., Emslie, G., Wells, K., Walkup, J. T., . . .Turner, J. B. (2009). The Treatment of Adolescent Suicide Attempters Study (TASA): Predictors of suicidal events in an open treatment trial. Journal of the American Academy of Child & Adolescent Psychiatry, 48, 987–996. doi:10.1097/CHI.0b013e3181b5dbe4

Carter, G., Reith, D. M., Whyte, I. M., & McPherson, M. (2005). Repeated self-poisoning: Increasing severity of self-harm as a predictor of subsequent suicide. British Journal of Psychiatry, 186, 253–257. doi:10.1192/bjp.186.3.253

Chanen, A. M., Jackson, H. J., McCutcheon, L. K., Jovev, M., Dudgeon, P., Yuen, H. P., . . .McGorry, P. D. (2008). Early intervention for adolescents with borderline personality disorder using cognitive analytic therapy: Randomised controlled trial. The British Journal of Psychiatry, 193, 477–484. doi:10.1192/bjp.bp.107.048934

Claes, L., Muehlenkamp, J., Vandereycken, W., Hamelinck, L., Martens, H., & Claes, S. (2010). Comparison of non-suicidal self-injurious behavior and suicide attempts in patients admitted to a psychiatric crisis unit. Personality and Individual Differences, 48, 83–87. doi:10.1016/j.paid.2009.09.001

Cotgrove, A. J., Zirinsky, L., Black, D., & Weston, D. (1995). Secondary prevention of attempted suicide in adolescence. Journal of Adolescence, 18, 569–577. doi:10.1006/jado.1995.1039

Csorba, J., Dinya, E., Plener, P., Nagy, E., & Páli, E. (2009). Clinical diagnoses, characteristics of risk behaviour, differences between suicidal and non-suicidal subgroups of Hungarian adolescent outpatients practising self-injury. European Child & Adolescent Psychiatry, 18, 309–320. doi:10.1007/s00787-008-0733-5

De Leo, D., & Heller, T. S. (2004). Who are the kids who self-harm? An Australian self-report school survey. Medical Journal of Australia, 181, 140–144.

Dougherty, D. M., Mathias, C. W., Marsh-Richard, D. M., Prevette, K. N., Dawes, M. A., Hatzis, E. S., . . .Nouvion, S. O. (2009). Impulsivity and clinical symptoms among adolescents with non-suicidal self-injury with or without attempted suicide. Psychiatry Research, 169, 22–27. doi:10.1016/j.psychres.2008.06.011

Elashoff, J. D. (2000). nQuery Advisor Version 4.0 user's guide. Los Angeles, CA: Statistical Solutions.

Evans, E., Hawton, K., Rodham, K., & Deeks, J. (2005). The prevalence of suicidal phenomena in adolescents: A systematic review of population-based studies. Suicide and Life-Threatening Behavior, 35, 239–250. doi:10.1521/suli.2005.35.3.239

Favazza, A. R., & Conterio, K. (1989). Female habitual self-mutilators. Acta Psychiatrica Scandinavica, 79, 283–289. doi:10.1111/j.1600-0447.1989.tb10259.x

Goodman, R. (2001). Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 1337–1345. doi:10.1097/00004583-200111000-00015

Gratz, K. L. (2001). Measurement of deliberate self-harm: Preliminary data on the Deliberate Self-Harm Inventory. Journal of Psychopathology and Behavioral Assessment, 23, 253–263. doi:10.1023/A:1012779403943

Grøholg, B., Ekeberg, O., & Haldorsen, T. (2000). Adolescents hospitalised with deliberate self-harm: The significance of an intention to die. European Child & Adolescent Psychiatry, 9, 244–254. doi:10.1007/s007870070027

Guertin, T., Lloyd-Richardson, E., Spirito, A., Donaldson, D., & Boergers, J. (2001). Self-mutilative behavior in adolescents who attempt suicide by overdose. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 1062–1069. doi:10.1097/00004583-200109000-00015

Hargus, E., Hawton, K., & Rodham, K. (2009). Distinguishing between subgroups of adolescents who self-harm. Suicide and Life-Threatening Behavior, 39, 518–537. doi:10.1521/suli.2009.39.5.518

Harrington, R., Kerfoot, M., Dyer, E., McNiven, F., Gill, J., Harrington, V., . . .Byford, S. (1998). Randomized trial of a home-based family intervention for children who have deliberately poisoned themselves. Journal of the American Academy of Child & Adolescent Psychiatry, 37, 512–518.

Hawton, K., Hall, S., Simkin, S., Bale, L., Bond, A., Codd, S., & Stewart, A. (2003). Deliberate self-harm in adolescents: A study of characteristics and trends in Oxford, 1990–2000. Journal of Child Psychology and Psychiatry, 44, 1191–1198. doi:10.1111/1469-7610.00200

Hawton, K., & Harriss, L. (2007). Deliberate self-harm in young people: Characteristics and subsequent mortality in a 20-year cohort of patients presenting to hospital. Journal of Clinical Psychiatry, 68, 1574–1583. doi:10.4088/JCP.v68n1017

Hawton, K., & Harriss, L. (2008). Deliberate self-harm by under-15-year-olds: Characteristics, trends and outcome. Journal of Child Psychology and Psychiatry, 49, 441–448. doi:10.1111/j.1469-7610.2007.01852.x

Hazell, P. L., Martin, G., McGill, K., Kay, T., Wood, A., Trainor, G., & Harrington, R. (2009). Group therapy for repeated deliberate self-harm in adolescents: Failure of replication of a randomized trial. Journal of the American Academy of Child & Adolescent Psychiatry, 48, 662–670.

Jacobs, D. G. (Ed.). (1999). The Harvard Medical School guide to suicide assessment and intervention. San Francisco, CA: Jossey-Bass.

Jacobson, C. M., Muehlenkamp, J. J., Miller, A. L., & Turner, J. B. (2008). Psychiatric impairment among adolescents engaging in different types of deliberate self-harm. Journal of Clinical Child and Adolescent Psychology, 37, 363–375. doi:10.1080/15374410801955771

Joiner, T. (2005). Why people die by suicide. Cambridge, MA: Harvard University Press.

Madge, N., Hewitt, A., Hawton, K., de Wilde, E. J., Corcoran, P., Fekete, S., . . .Ystgaard, M. (2008). Deliberate self-harm within an international community sample of young people: Comparative findings from the Child & Adolescent Self-harm in Europe (CASE) Study. Journal of Child Psychology and Psychiatry, 49, 667–677. doi:10.1111/j.1469-7610.2008.01879.x

Muehlenkamp, J. J. (2005). Self-injurious behavior as a separate clinical syndrome. American Journal of Orthopsychiatry, 75, 324–333. doi:10.1037/0002-9432.75.2.324

Muehlenkamp, J. J., & Gutierrez, P. M. (2004). An investigation of differences between self-injurious behavior and suicide attempts in a sample of adolescents. Suicide and Life-Threatening Behavior, 34, 12–23. doi:10.1521/suli.34.1.12.27769

Muehlenkamp, J. J., & Gutierrez, P. M. (2007). Risk for suicide attempts among adolescents who engage in non-suicidal self-injury. Archives of Suicide Research, 11, 69–82. doi:10.1080/13811110600992902

National Institute for Health and Clinical Excellence. (2004). Self-harm: The short-term physical and psychological management and secondary prevention of self-harm in primary and secondary care (Clinical Guideline 16). London, England: Author. Retrieved from http://www.nice.org.uk/nicemedia/pdf/CG016NICEguideline.pdf

Nock, M. K., Borges, G., Bromet, E. J., Cha, C. B., Kessler, R. C., & Lee, S. (2008). Suicide and suicidal behavior. Epidemiologic Reviews, 30, 133–154. doi:10.1093/epirev/mxn002

Nock, M. K., Joiner, T. E., Jr., Gordon, K. H., Lloyd-Richardson, E., & Prinstein, M. J. (2006). Non-suicidal self-injury among adolescents: Diagnostic correlates and relation to suicide attempts. Psychiatry Research, 144, 65–72. doi:10.1016/j.psychres.2006.05.010

Nock, M. K., & Kessler, R. C. (2006). Prevalence of and risk factors for suicide attempts versus suicide gestures: Analysis of the National Comorbidity Survey. Journal of Abnormal Psychology, 115, 616–623. doi:10.1037/0021-843X.115.3.616

Office of Population Censuses and Surveys. (1991). Census user guide: 44. Social class based on occupation: Definitions in terms of standard occupational classification (SOC) unit groups and employment status. London, England: Author.

Ougrin, D., Banarsee, R., Dunn-Toroosian, V., & Majeed, A. (2010). Suicide survey in a London borough: Primary care and public health perspectives. Journal of Public Health. Advance online publication. doi:10.1093/pubmed/fdq094

Ougrin, D., & Latif, S. (2011). Engagement with specific psychological treatment versus treatment as usual in adolescents with self-harm: Systematic review and meta-analysis. Crisis: The Journal of Crisis Intervention and Suicide Prevention, 32, 74–80. doi:10.1027/0227-5910/a000060

Ougrin, D., Ng, A. V., & Low, J. (2008). Therapeutic assessment based on cognitive-analytic therapy for young people presenting with self-harm: Pilot study. Psychiatric Bulletin, 32, 423–426. doi:10.1192/pb.bp.107.018473

Ougrin, D., Zundel, T., Ng, A., Banarsee, R., Bottle, A., & Taylor, E. (2011). Trial of therapeutic assessment in London: Randomised controlled trial of therapeutic assessment versus standard psychosocial assessment in adolescents presenting with self-harm. Archives of Disease in Childhood, 96, 148–153. doi:10.1136/adc.2010.188755

Pattison, E. M., & Kahan, J. (1983). The deliberate self-harm syndrome. The American Journal of Psychiatry, 140, 867–872.

Posner, K., Oquendo, M. A., Gould, M., Stanley, B., & Davies, M. (2007). Columbia Classification Algorithm of Suicide Assessment (C-CASA): Classification of suicidal events in the FDA's pediatric suicidal risk analysis of antidepressants. The American Journal of Psychiatry, 164, 1035–1043. doi:10.1176/appi.ajp.164.7.1035

Poston, J. M., & Hanson, W. E. (2010). Meta-analysis of psychological assessment as a therapeutic intervention. Psychological Assessment, 22, 203–212. doi:10.1037/a0018679

Ross, S., & Heath, N. (2002). A study of the frequency of self-mutilation in a community sample of adolescents. Journal of Youth and Adolescence, 31, 67–77. doi:10.1023/A:1014089117419

Ryle, A. (2010). Cognitive analytic therapy. In N.Kazantzis, M. A.Reinecke, & A.Freeman (Eds.), Cognitive and behavioral theories in clinical practice (pp. 244–277). New York, NY: Guilford Press.

Sarkar, P., Sattar, F. A., Gode, N., & Basannar, D. R. (2006). Failed suicide and deliberate self-harm: A need for specific nomenclature. Indian Journal of Psychiatry, 48, 78–83. doi:10.4103/0019-5545.31594

Schmidtke, A., Bille-Brahe, U., Deleo, D., Kerkhof, A., Bjerke, T., Crepet, P., . . .Sampaio-Faria, J. G. (1996). Attempted suicide in Europe: Rates, trends and sociodemographic characteristics of suicide attempters during the period 1989–1992. Results of the WHO/EURO Multicentre Study on Parasuicide. Acta Psychiatrica Scandinavica, 93, 327–338. doi:10.1111/j.1600-0447.1996.tb10656.x

Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, H., & Aluwahlia, S. (1983). A children's global assessment scale (CGAS). Archives of General Psychiatry, 40, 1228–1231.

Silverman, M. M., Berman, A. L., Sanddal, N. D., O'Carroll, P. W., & Joiner, T. E. (2007). Rebuilding the tower of Babel: A revised nomenclature for the study of suicide and suicidal behaviors. Part 1: Background, rationale, and methodology. Suicide and Life-Threatening Behavior, 37, 248–263. doi:10.1521/suli.2007.37.3.248

Stanley, B., Winchel, R., Molcho, A., & Simeon, D. (1992). Suicide and the self-harm continuum: Phenomenological and biochemical evidence. International Review of Psychiatry, 4, 149–155. doi:10.3109/09540269209066312

TADS Team. (2007). The Treatment for Adolescents With Depression Study (TADS): Long-term effectiveness and safety outcomes. Archives of General Psychiatry, 64, 1132–1143. doi:10.1001/archpsyc.64.10.1132

Walsh, B. W. (2006). Treating self-injury: A practical guide. New York, NY: Guilford Press.

Walsh, B. W., & Rosen, P. M. (1988). Self-mutilation: Theory, research, and treatment. New York, NY: Guilford Press.

Wichstrøm, L. (2009). Predictors of non-suicidal self-injury versus attempted suicide: Similar or different?Archives of Suicide Research, 13, 105–122. doi:10.1080/13811110902834992

Wong, P. S. J. (2007). Self-injurious behaviors in Hong Kong adolescents: Cross sectional and prospective studies. Hong Kong, People's Republic of China: University of Hong Kong.

Wood, A., Trainor, G., Rothwell, J., Moore, A., & Harrington, R. (2001). Randomized trial of group therapy for repeated deliberate self-harm in adolescents. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 1246–1253. doi:10.1097/00004583-200111000-00003

World Health Organization. (1992). The ICD–10 classification of mental and behavioural disorders: Clinical descriptions and diagnostic guidelines. Retrieved from http://www.who.int/classifications/icd/en/bluebook.pdf

Submitted: August 14, 2010 Revised: May 13, 2011 Accepted: June 1, 2011